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ON ESTIMATION OF PARAMETERS IN LATENT STRUCTURE 
ANALYSIS 


T. W. ANDERSON* 


COLUMBIA UNIVERSITY 


The latent structure model considered here postulates that a population 
of individuals can be divided into m classes such that each class is “homogene- 
ous” in the sense that for the individuals in the class the responses to N 
dichotomous items or questions are statistically independent. A method 
is given for deducing the proportions of the population in each latent class 
and the probabilities of positive responses to each item for individuals in 
each class from knowledge of the probabilities of positive responses for 
individuals from the population as a whole. For estimation of the latent 
parameters on the basis of a sample, it is proposed that the same method 
of analysis be applied to the observed data. The method has the advantages 
of avoiding implicitly defined and unobservable quantities, and of using 
relatively simple computational procedures of conventional matrix algebra, 
but it has the disadvantages of using only a part of the available information 
and of using that part asymmetrically. 


1. Introduction 


Latent structure analysis is designed for investigations in which the 
fundamental data consist of sets of discrete statistical variables. The case of 
particular interest to social psychologists and sociologists is that in which 
each of a number of individuals responds positively or negatively to each of a 
number of items. In particular, these may be answers to dichotomous questions 
in a questionnaire. The analysis is based on a mathematical model for which 
the principal assumption is that the population may be classified into homo- 
geneous sets such that in each set the responses on different items are inde- 
pendent in the probability sense. If the number of homogeneous sets is 
finite, we have what is called the latent class model. This model which is 
treated in this paper is given explicitly in the next section. 

P. F. Lazarsfeld has written about latent structure analysis in Chapters 
10 and 11 of (1). He and collaborators have also written a series of (mimeo- 
graphed and dittoed) papers, ““The Use of Mathematical Models in the 
Measurement of Attitudes,” issued by the RAND Corporation (2). In these 
papers the rationale and purpose of the model are given as well as examples 
of the application of the analysis to data in psychology and sociology. The 
reader of this journal is also referred to the article by Green (3). 

In an unpublished memorandum Lazarsfeld has indicated how the 
“latent parameters” corresponding to a particular item can be obtained 
from knowledge of the joint probabilities of the responses. The method is to 

*Work supported by the RAND Corporation. 
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find the roots of a certain determinantal equation. In this paper the method 
is extended to obtain simultaneously the latent parameters corresponding to 
many items and the “‘latent class frequencies.’’ The extension of the method 
involves the use of matrix algebra; in particular, use of vectors associated 
with the roots of the determinantal equations leads to the other parameters. 

To use a sample to estimate the parameters of the model it is proposed 
to apply the same methods to the relative frequencies observed in a sample. 
A numerical example is given which shows the computations to be performed. 
In another paper it will be shown that these estimates have an asymptotic 
joint normal distribution as the sample size increases. The limiting means 
are the parameters estimated; the limiting variances and covariances are 
functions of the underlying probabilities of responses, which can be estimated 
consistently from the sample. These results can be used in the usual way to 
obtain confidence regions and tests of hypotheses. 

Green (3) has given another method for estimating these parameters. 
The two methods are based on essentially the same algebraic analysis of the 
latent structure model. Green’s method, however, uses more of the data from 
the sample and uses it in such a fashion that all items are treated in the same 
way. It would appear that in many cases Green’s method or some minor 
modification of it (e.g., replacing P, by other linear combinations of P, in 
Green’s notation) would be more efficient than the method presented in this 
paper. 

An advantage of the method presented here over Green’s method is the 
simplicity of computation; in fact, these computations can be done me- 
chanically (for example, on punched-card equipment). Green’s method has 
the inherent difficulty that in the matrices of data to be used certain elements 
cannot be observed and these elements must be approximated in some way; 
in the present method this difficulty is avoided by treating the data asym- 
metrically. 

Another reason for studying the present method is that the availability 
of large sample distribution theory makes possible obtaining standard errors 
of estimate, confidence intervals, and tests of hypotheses and thereby gives 
methods for studying the design of models for getting the best determination 
of the parameters. Some studies of this sort are now being made; these studies 
together with the analysis of some sampling experiments give suggestions 
for combining various estimates of the type treated here. 

It seems reasonable to expect that neither the estimation method of 
Green nor that of this paper yields efficient estimates in the sense of estimates 
with minimum asymptotic variance. To obtain efficient estimates one would 
try the method of maximum likelihood, minimum x’, or another of what 
Neyman calls Best Asymptotic Normal estimation procedures (4). The 
methods of Green and this paper could be useful as initial approximations in 
an iterative method leading to efficient estimates. 
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2. The Latent Class Structure 


We assume that individuals respond positively or negatively to each 
of a set of K items. The probability of drawing an individual from the popula- 
tion and obtaining from him a positive response on item 7 is denoted by 7; 
(a negative response by 7;.), positive responses on items 7 and j by 7;; (a 
positive response on item 7 and a negative one on item j by 7;;.), etc. The 
basic probabilities are the 2“ with K subscripts m12...« , , 
the x’s with less than K subscripts are linear combinations of them. Since 
there are K dichotomous items there are 2“ different possible response 
patterns. These represent 2“ mutually exclusive, and exhaustive events, and, 
therefore, the 7’s define a multinomial distribution for these 2“ categories. 

In the latent class model it is assumed that the population can be divided 
into m (mutually exclusive) “homogeneous” subpopulations or classes. Let 
v* be the probability that a person drawn at random is drawn from the 
a-th subpopulation (2 = 1, --- , m). Let 4% be the probability that an 
individual from the a-th subpopulation responds positively to item 7 (AZ = 
1 — A% that he responds negatively). The crucial assumption of latent 
structure analysis is that the probability of a collection of responses of an 
individual from a given class is the product of the probabilities of the re- 
sponses to individual items of the collection. For example, the probability 
for an individual in class @ of positive response to items i and j is \‘A4. For 
an individual in a given class, therefore, the responses to the various items 
are statistically independent. 

In this model the z’s are functions of the v’s and }’s, for the probability 
of drawing an fhdividual and getting a certain response pattern is the sum 
over the latent classes of the probability of being drawn from a class multi- 
plied by the probability for an individual in that class of giving the response 
pattern. For instance 


(2.1) 
Tie = DO NADAS, 


= DO tj, k 6, 


The class to which an individual belongs cannot be observed directly; 
it can be inferred only from the response pattern which he gives. In order to 
make this inference the investigator must use the parameters of the latent 
structure, that is, the v’s and )’s, together with the observed response pattern 
of each individual. 

In principle the investigator can determine directly (i.e., by means of 
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an infinitely large sample) the underlying probabilities of responses, the 
n’s£Our problem is to use the z’s to obtain the »’s and )’s. Essentially this 
means solving equations (2.1) (extended to z’s with all possible subscripts). 
We shall give a method for doing this. The method of statistical estimation 
proposed here is the same but applied to estimates of the z’s. 

Let 


A= (2.2) 


be the matrix of latent parameters. The superscript refers to the row and 
denotes the class; the subscript refers to the column and denotes the item. 
It will be convenient to refer to the a-th element in the first column as 
\% (=1) and consider it as referring to a dummy item that always has a 
positive response. 


Let 
v 0 0 
(2.3) 


Let A, be the m X m matrix consisting of the first m columns of A, and let 
A, be the m X m matrix consisting of the first column and the next m — 1 


columns of A not contained in A, [i.e., the first row of A, is (1 An «++ am-2)]. 
Let 


where k > 2m — 2. Since the initial numbering of the items is arbitrary, it 
is seen that we have selected from all of the items two sets of m — 1 items 
and one additional item. This implies that the battery contains at least 
2m — 1 items. (Other methods may not necessarily require so many items.) 

To deduce the selected \’s and »’s we shall use the equations of (2.1) 
for and z;;,7 = 0,1, 2,---,m—1landj =0,m,m+1, , 2m — 2. 
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When a subscript is 0, the subscript is suppressed; for example, mj, = jx , 
Took = Tor = TT; - We also define To00 = To = TM = 1. Let 


Took Tomk T0,m+1,k 0,2m-2,k 
Tiok Timk T1,m+1,k 11,2m-2,k 
II = | Toamk T2,m+1,k W2,2m-2,k 


LTm-1,0k Wm-1,mk Wm-1,m+1,k Wm-1,2m-2,k 


Tik Timk T1,m+1,k ,2m-2,k 
Tomk T2,m+1,k %2,2m-2,k (2.4) 


L_Tm-1,k Tm-1,mk Wm-i,mt+1,k Wm-1,2m-2,k 


The subscript & is on each element of II, the row subscripts (the first sub- 
scripts when not suppressed) correspond to the items in A, and the column 
subscripts (the second subscripts when not suppressed) correspond to the 
items in A, . Finally let II* be defined the same as II except that each element 
™;;, is replaced by 2,;; . The fundamental equations are 


A‘NAA,, = A‘NA,. (2.5) 
These can easily be verified from (2.1). 
The roots 6’, --- , 6” of the determinantal equation 
— | =0 (2.6) 


can be determined from the probabilities of responses. This equation is 


| 1 — | = | AJNAA, — BANA, | 
= | AIN |-| 4 — 61 |-| As | (2.7) 
| Ar N |-| — |-| As |. 


0 


Since A is a diagonal matrix, the roots of (2.7) are these diagonal elements, 
namely, \; , -:* , \%7. It should be noticed that we require A, , N, Az to be 
nonsingular; thus every v“ must be different from zero. 

We shall now show how knowledge of the 6’s together with II and II* 
can be used to find A, , A, , and N. 
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Since | II — 0@°II* | = 0, the matrix (II — 6°II*) is singular and it is 
possible to find a (column) vector x“ (not all components zero) satisfying 
(II — 6° II*)x” = 0, that is, 


= (2.8) 


If the roots 6’, --- , 6” are different, x* is uniquely determined except for a 
multiplicative constant; that is, since (II — @“II*) is of rank m — 1, there is 
one linearly independent solution of (2.8). Let X = (2° --- 2”) and Oa 
diagonal matrix with 6“ as the a-th diagonal element. Then if we set side by 
side the m equations (2.8) we have 


IX = (2.9) 


Suppose that the 6’s in © are ordered so that 6 = A, the diagonal matrix 
appearing in (2.7). Then one solution of (2.9) for X is X = Az’. This is 
easily verified by replacing II and II* in (2.9) by (2.5), © by A, and X by 
, obtaining 


= ALNA,A;"A. (2.10) 


When the 6’s are ordered, X is uniquely determined except for multiplica- 
tion on the right by an arbitrary (nonsingular) diagonal matrix (i.e., multipli- 
cation of each column of X by a constant). Thus any solution X can be 
expressed as X = A;'E, , where E is a diagonal matrix. Conversely given a 
solution X, we have A, = E,X~*. Since the elements of the first column of 
A, must be unity, each diagonal element of E, must be the reciprocal of the 
first element of the corresponding row of X~’. Thus the right-sided character- 
istic vectors of II in terms of II* determine A, . 

Now consider the row vector (y*)’ satisfying (y*)/(II — 6°II*) = 0. 
Transposition gives 


Ii’y* = (2.11) 


where II’ = ASNAA, and II*’ = AjNA, . This is the same as (2.8) with 2” 
replaced by y*, and A, and A, interchanged. The argument used previously 
shows that A, = E,Y~*, where Y = (y' --- y”) is a set of vectors satisfying 
(2.11) and E, is a diagonal matrix in which each diagonal element is the 
reciprocal of the first element in the corresponding row of Y~’. 

Finally, from (2.5) we obtain 


(AD = (Af) = N. (2.12) 


We assumed earlier that the 6’s were numbered to correspond to the 
\% . This is the only arbitrariness in our solution. It is obvious, however, 
from the symmetry of (2.1) that this is inherent in the model. 

We can avoid the inversion of matrices by noting that 
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= (A{NA,)(A;'E.) = A{NE, , (2.13) 
= (AJNA,)(A;'E,) = ASNE, , (2.14) 
ALY = A,A;'E, = E, = N7"(NE,). (2.15) 


Since the first row of A{ consists of 1’s, the elements of the first row of II*X 
are the diagonal elements of NE, ; if we divide each column of II*X by its 
leading element we obtain Aj . Similarly from II*’Y we obtain NE, and 
Aj. Finally from A,Y and NE, we find N“* and hence N. 


3. Estimation of Parameters 


The latent class model is to be used when the investigator has drawn a 
sample of dichotomous response patterns and has reason to believe that 
the joint probabilities have the structure indicated by equations (2.1). The 
investiga’ or will estimate the x’s by the corresponding p’s, which are the 
observed relative frequencies. For example, the estimate of 7, is p; , the pro- 
portion of respondents giving a positive response to the 7-th item. Now the 
question is how to estimate the )’s and »’s from the p’s. 

It is proposed here to estimate these by applying the results of Section 
2 to the p’s. In this way we estimate the probabilities of the latent classes 
and the probabilities associated with 2m — 1 items by treating 2m X m 
matrices of p’s. To obtain estimates of the probabilities associated with 
other items we must use other pairs of m X m matrices. We presume that the 
investigator knows what m is. If he does not, he can assume a value for m 
and compare the observed p’s with up to K subscripts with the estimates of 
the z’s computed on the estimated )’s and v’s. 

It is clear that in use of this method of estimation, there is a choice of 
which matrices to use for estimating particular parameters. In the population 
it did not matter which 7’s were used for deriving the v’s and )’s (as long as 
the matrices were nonsingular) because the assumption of the model implies 
that the set of equations is consistent in the sense that any set of equations 
that can be solved will give the same v’s and }’s. In the sample, however, the 
values of the estimates of v’s and )’s will depend on the p’s used (because the 
p’s will not usually satisfy the conditions of algebraic consistency). This 
raises questions as to the choice of p’s and as to how to combine different 
estimates of the same parameters. The asymptotic distribution theory 
developed elsewhere suggests answers to these questions; roughly speaking 
the larger the determinants of A, and A, are the better will be the estimates. 


4. An Example 


For a numerical example of the proposed method we have used some 
artificial data. The RAND Corporation has constructed observations from a 
latent structure by using random numbers in conjunction with a theoretical 
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structure. We give in the table below the probabilities of the latent classes 
and the probabilities of positive response on several items by individuals in 
the different latent classes. From 4000 experimental observations, four 
samples of 1000 each were constructed. We use one of these samples for 
illustrative purposes. Of course, such an example avoids many of the diffi- 
culties of working with actual data (such as determining the appropriate 
number of classes), but it has the obvious advantage of focussing attention 
on the computation. 


THEORETICAL STRUCTURE 


Probability Probability of Positive Response 
of Class Item 2 Item 6 Item 3 Item 7 Item 1 
Class 1 3 9 2 8 4 9 
Class 2 5 i7 9 4 8 5 
Class 3 2 1 3 3 


We give below the observed relative frequencies of positive responses on 
single items, on pairs of items and on triples of items, the triples all including 
Item 1. 


FREQUENCIES OF POSITIVE RESPONSES 


Single Pairs of Items 
Items 4 6 3 7 1 
Item 2 .639 .379 .346 .409 .403 
Item 6 .544 .379 .410 .278 
Item 3 .483 .346 .221 oo .278 .313 
Item 7 .586 .409 .410 .278 .319 
Item 1 .530 .403 .278 .313 .319 sae 
Item 1 and Pairs of Items 
2 6 3 7 
2 .198 {252 . 242 
6 .198 wan .120 .207 
3 252 .120 .173 
7 . 242 . 207 .173 


For instance 639 out of 1000 responded positively to Item 2, 346 out of 1000 
responded positively to both Items 2 and 3, and 252 out of 1000 responded 
positively to Items 1, 2, and 3. The numbers of the items are those of the 
original study, but we have ordered them 2, 6, 3, 7, 1 because we shall con- 
sider the parameters of Items 2 and 6 as constituting A, , Items 3 and 7, 
A, , and Item 1, A. (In other words we should have renumbered the items to 
fit the notation of Section 2.) 
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For estimation we use the sample equivalents of II and II*, namely, 


.313 1.000 .483 .586 
403 .252  .242 |, P* =| .639 .409 |. (4.1) 
.278 .120 .207 544 .221 .410 


First of all we find the roots of the determinantal equation 
.503 — 1.000¢ .313 — .483¢ .319 — .586¢ 

|P — tP*| =| 403 — 639% .252 — 346¢ .242 — 409¢|=0. (4.2) 
278 — .544¢ .120 — .221¢ .207 — .410¢ 


‘The determinant can be expanded to give a polynomial of third degree. The 
roots are t' = .93661, = .49179, and ¢® = .12592. In the matrix in (4.2) 
we now replace ¢ by ¢’ to give 


— 40661 —.13938 —.22985 
P — =| —.19549 —.07207 —.14107 |. (4.3) 
— .23152 —.08699 —.17701 


This matrix is singular. We take the set of cofactors of any row to be the 
elements of the estimate of the vector x’, say 4’; if we use cofactors of the 
first row, the components of this vector are —.07207 X (-—.17701) — 
(—.08699)(—.14107) = .000,485,43, —.001,943,16, and .000,320,03. Since a 
constant of proportionality is arbitrary, we multiply by 10*. In turn we replace 
t by ? and @ and from each matrix use cofactors of the first row to give £” 
and Thus 


4.8543 — 2.2646 148.2538 
X = (4'44°) =| —19.4816 — 4.8783 —102.0329 |. (4.4) 
3.2003 14.6898 —139.3757 


We then compute the matrix P*X. The elements in the first row of P*X 
are —2.65579, 3.98740, 17.29775; these are the diagonal elements of NE, 
We divide each element of the first column of P*X by —2.65579, etc., to 
obtain the estimate of A, , namely, 


1.00000 1.00000 1.00000 
Li =| .87074 .72056 .14026 |. (4.5) 
12859 .05531 


We now use the transpose of (4.3) [that is, (P — ¢’P*)’] to find the first 
column of Y (whose elements are 4.8543, —46.7700, 30.9705), and similarly 
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we find the other columns of Y. The first row of sat (namely, —8.18378, 
24.68949, 25.61189) gives the diagonal elements of NE,. The estimate of 
Ad is 
1.00000 1.00000 1.00000 
[3 =| .85453 .37933 .29115 |. (4.6) 
43823. .78576 ~—-.25307 


Next we compute 
3.89640  .00074 —.00248 
L,Y(NE,)"' =| .00108 1.86281 .00816 |. (4.7) 
—.00092 .00195 4.81361 


This should be a diagonal matrix; the size of the nondiagonal elements 
indicates the accuracy of the computation. We take the diagonal elements 
to be the estimates of the diagonal elements of N~'. The reciprocals of the 
diagonal elements of (4.7) are the estimates of the diagonal elements of N, 
namely, .25665, .53682, .20774. The estimated structure is given in the 
following table. This is to be compared with the theoretical structure given 
earlier. 


ESTIMATED STRUCTURE 


Probability Probability of Positive Response 
of Class Item 2 Item 6 Item 3 Item 7 Item 1 
Class 1 .257 .871 .129 .855 . 438 .937 
Class 2 .537 .931 .379 . 786 .492 
Class 3 . 208 .140 .055 .291 . 253 .126 
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The personnel classification problem is identified mathematically with 
other problems in the social and biological sciences. This mathematical 
problem is shown to be a special case of the general mathematical problem 
of linear programming. It is proposed here that the personnel classification 
problem may be solved directly by methods particularly appropriate to it 
as well as by the simplex method, which is a standard method for solv. g the 
general linear programming problem. The method of optimal regions is de- 
rived and illustrated in this paper. 


1. Introduction 


The general type of problem considered in this paper is first illustrated 
with a trivial problem. A business concern needs to fill three jobs which 
demand different abilities and training. Three applicants who can be hired 
for identical salaries are available. Because of the different abilities, training, 
and experiences, however, the value of each applicant to the company depends 
upon the job in which he is placed. The estimate of the value of each applicant 
to the company each year if he were to be assigned to any one of the three 
jobs is given in Table 1. 

It is desired to assign the applicants to the jobs in such a way that the 
total value to the company is as great as possible. In this problem there are 
3! = 6 possible alternative assignments of the three men and the greatest 
estimated value to the company, $24,000 per year, is obtained by assigning 
applicant 1 to job 3, applicant 2 to job 1, and applicant 3 to job 2. 

A more general problem is arrived at by denoting by c;; the contribution 
of individual 7 to the common effort if he is assigned to job 7, in units of the 
measure of the common effort. Thus in Table 1, ¢c,, = 5, ci. = 4, G3 = 7, 
C2, = 6, etc. Sometimes dimensionless c;; may be used to express the relative 
contributions to the common effort. 

A more general problem features N individuals and N jobs in which, as 
above, the contribution of individual 7 to the common effort is c;; units if he 


*Much of the basic research covered in this paper was carried out while the author 
was working on the problem of personnel classification in his capacity as Consultant, 
Personnel Research Branch, Adjutant General’s Office, Department of the Army. Some 
of the material was presented in a conference which was held at Personnel Research Branch 
in August, 1952. The author wishes to express his appreciation to the Department of the 
Army for permission to use these materials in this paper. The opinions expressed are 
those of the author and are not to be construed as official or as those of the Department 
of the Army. 
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is assigned to job j. The assignments are then made so that the sum of the 
corresponding c;; values is a maximum. We set 2z;; = 1 when individual i 
is assigned to job j and z;; = 0 otherwise. Then 


and the problem is to maximize* 


= Lili; (1.2) 


There are N! values of 7, though some of them may be equal, and for 
N large it becomes impractical to examine them all to determine the maximum. 
Improved methods of solution are needed. 

Frequently there are many identical jobs or jobs which, though not 
identical, demand the same basic qualifications indicated by the c;; . Such 
jobs can be combined into a job category. If there are m categories and if 
the number of jobs grouped in the jth job category is qg; , we have 


1 if 7 is assigned to j (1.3) 


x;; = 0 otherwise, 


and 


| 
with = Dg = Di=N. (14) 

j=1 

We wish to maximize the total contribution to the common effort 


= - (1.5) 


The g; values are called quotas. This formulation of the problem is illustrated 
in Table 2. 

Sometimes there are different individuals having identical c;; values or 
at least the values are close enough so that the individuals may be grouped 
without serious error. In this case we group these individuals into personnel 
categories. If there are n personal categories and the number of individuals 
grouped in the ith personnel category is f; , we have 


1 if some individual in 7 is assigned to 7 (1.6) 


x;; = 0 otherwise, 


*In certain situations, particularly when the smaller values of c,, indicate the greater 
contributions to the common effort, the values of 7’ should be minimized. 


| 
| 
| 
| 
| 
v 
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TABLE 1 


TABLE 2 


Use of Job Categories 


Identification of Personnel Classification 


with Linear Programming 


General | Personnel | Personnel] Personnel | Personnel 
(General- | (Table 2) (Table 3) | (Table k) 
ization off 
Table 1) 

i i  E i i 

j J j 

n n2 Nm Nn mn 

Cy Cij ij 

bj; qj or lor f; or Qj 

1 or 0 loro lor k or 0 


Estimated Value Per Year i 1 2 3 J 
eee eee 1 
in Units of $1000 1 1m 
2 Co, Cop Com 1 
Job 3 £30 £33 eee eee 1 
2 3 eee eee eee 
eee eee 1 
% 4 1% “2 ij Sim 
Applicant] 2 6 7 3 
92 eee cee Om N 
3 TABLE 
Use of Personnel Categories Use of Personnel Categories and Job Categories 
j 
c c c sos ¢ C f 
2 C21 €22 vee CoN fo 23 *2 
eee eee fy il ie B ij imj “2 
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and 


= 1 
1=1 
N 
= fi 
The values of f,; are called frequencies. This formulation of the problem 


is illustrated in Table 3. 
A common form of the problem uses both personnel categories and job 
categories. If, as before, f; is the number of persons in personnel category 7 


and q; is the number to be placed in job category j, then 
Z,;; = zero or a positive integer indicating the number of persons (1.8) 


in personnel category 7 assigned to job category /. 


Then, 


With = Da =N. (19) 
tii = | 
This formulation of the problem is illustrated in Table 4. 

Table 3 is a special case of Table 4 when g; = 1; Table 2 is a special case 
of Table 4 when f; = 1, and the square N by N matrix results when f; = 1 
and gq; = 1. 

The problem under discussion is closely related to problems previously 
discussed in Psychometrika by Brogden (1), Thorndike (9), Votaw (11) and 
Lord (5). The formulation of Table 2, used essentially by Brogden (1, 149), 
Thorndike (9, 233), and Lord (5, 297), appears to be a natural form of the 
personnel classification problem in which several or many men are to be 
assigned to a relatively small number of job categories. The formulation in 
Table 4 has been used by Votaw (11, 257) in identifying the problem as a 
linear programming problem. 


2. Equivalent Mathematical Problems 


This problem is essentially the equivalent, mathematically, of problems 
arising in other applied fields. For example, it is similar to the Hitchcock 
transportation problem, solutions and theory of which have been supplied 
by many, including Dantzig (3) and Flood (6). This calls for the shipping of 
z,; unit parcels from origin 7 to destination j when a, parcels are to be shipped 
from origin 7, b; parcels are to be delivered to destination j, so that the total 
transportation cost >>, ; z,,¢,; isto be a minimum if ¢,; is the cost of shipping 
a unit parcel from origin 7 to destination 7. Equations (1.8), (1.9), and Table 


: i 
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4 are immediately applicable to this problem if f; is replaced by a; and q; 
by b; . 

Rao (7, 322-329) has shown how different problems in biometric classi- 
fication reduce to the mathematical problem discussed above. In one of these, 
for example, it is desired to classify an individual into one of m classes on the 
basis of the measurements of the individual. We desire to minimize the total 
risk when c;; is the expected risk of erroneous classification resulting from 
placing individual 7 in class 7 when he does not belong there. The equations 
(1.6), (1.7), and Table 2 are applicable. 

Also, von Neumann (10) has related the personnel classification problem 
to a certain zero-sum two-person game. 

These problems and other similar ones are all special cases of the general 
problem of linear programming. Dantzig (2) has presented a mathematical 
model which is applicable to many problems, after reduction, in linear pro- 
gramming as follows: 


Find the values of \, , 42, -** , A, Which maximize the linear form 


+ + + Aden 
subject to the conditions that 4; > 0,7 = 1, 2, ---, m, and 


+ + + Andon = be 


+ + + Aion = bn 
where a;; , b; , are constants (i = 1,2, ,m;7j = 1, 2, ---, n). 


The personnel classification problem is a special case of this problem. 
The correspondence between the general problem and (a) the problem ex- 
pressed in the individual form of which Table 1 is a special case when N = 3, 
(b) the forms of Tables 2, 3, and 4 is indicated in Table 5. 

Since the classification problem, which is hereafter used to designate the 
mathematical problem (including minimization) encompassing the alternative 
problems described above, is a special case of linear programming, it follows 
that methods of solution of the general linear programming problem are 
available. In particular the simplex method is available. This has been pointed 
out by Votaw (11) to be a solution of the classification problem and furthermore 
a solution that in some cases can be carried out with a high speed computing 
machine. Even though modifications of the simplex method, as applied to the 
classification problem, are possible, this is not the only useful method avail- 
able. It is the purpose of this paper to introduce the method of optimal 
regions and to show how it can be used. 


| 
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It does not appear at first that the personnel classification problem is 
considerably simpler, mathematically, than the general linear programming 
problem. However, examination of the last line of Table 5 shows that the 
a;; are integral. This fact seems to indicate a much simpler solution than is 
possible with the less restricted a;; , for example, those which usually appear 
in game theory. The implication is that we should solve the classification 
problem directly from the mathematical model rather than to use a method of 
solution which is designed to handle the more general problem. 


3. Different Forms of the Problem 


Tables 1, 2, 3, and 4 illustrate four different forms in which the problem 
may appear. We may call the form of Table 1, which features an N by N 
matrix, the individual form since neither individuals nor jobs are grouped; 
the form of Table 2, the quota form since the quotas for each job category are 
specified; the form of Table 3, the frequency form since the frequencies of the 
personnel categories are specified; and the form of Table 4, the frequency- 
quota form. Actually the frequency form appears seldom in classification 
problems so we can drop it from further consideration and use the term 
“frequency form’’ to described the situation in Table 4. In fact, this frequency 
form is usually not the natural form of the classification problem but results 
from the grouping of individuals which is demanded when the problem is to 
be worked by the previously available, simplex method. 

Each of these forms describes a situation in which the number of men is 
identical with the number of jobs. Alternative situations can be reduced 
formally to the forms above with the introduction of dummy men, or dummy 
jobs, having c;; = 0. This is illustrated in Table 6, where the c,; values for 
ten men and two job categories are given. In problem (a) the quotas are 
gq: = 4and gq, = 4. It is sufficient to have a dummy job category with q,; = 2 
to complete the assignment. In problem (b) the quotas are gq, = 6 and q, = 6. 
Two dummy men are introduced. 

Occasionally a problem arises in which no quotas are specified. The solu- 
tion in this case is extremely simple. Assign each person according to his job 
profile, i.e., to the job category for which his c;; value is largest. This solution 
gives the greatest possible 7, irrespective of quotas, and is sometimes in- 
cluded with that of the quota solution in order to show the effect of the quota 
restrictions. 


4. The Case of Two Job Categories 


There is no assignment problem when there is only one job category. 
For the case of complete quotas all individuals are assigned to this job cate- 
gory and, for the case of partial quotas, the individuals are assigned in the 
decreasing order of the c;, values. 

The problem really begins with two job categories. As was pointed out 


TABLE 
TABLE 6 Subtraction ofk from cya 
4 6 
(a). Personnel Surplus (bv). Personnei Deficiency lel 
4 2 qj 6 6 
4] 26 2 0 4] 26 2 3 
cre 5 | 26 25 20 1 1 1 
6129 27 ° 6 | 29 27 4 | 26 = 22 1 1 1 1 
1 1 i? 1-2 
7/29 30 0 2 29 2 
67/25 33 8 | 25 33 2 : : 
2 3 
9/25 37 0 3| 2 /e 
30 49 10 | 30 9 10] 30 tg | 2 
= 
12 0 ° cy 0 3 
cy 308 
cy 303 
TABLE 7 
Use of Differences 
TABLE 9 
a] 6 
Individual Form with Optimal Regions 
[ei] ese | cas | | J | Unite of $1000 Per Year 
1 | 23 ] 13 10 1] 23 7 1} 16 1 2 
2 | 23 | 27 6 13/23 3 1] 20 
4 30 | 20 10 1] 30 1 1] 23 43 > | 2 1 1 
26 | 22 4 1] 26 1 1} 25 rl 
5 | 28 | 25 3 2} 25 2] 28 
6 | 29 | 27 2 2| 27 =k 2] 30 
7 | 29 | 30 “1 | 2] 30 2] 33 1 5 T 
8 | 25 | 33 2] 33 -11 2 2 6 7 3 
9 | 37 -12 2] 37 #15 2 3 2:8. 2 
TABLE 10 
Quota Form with Optimal Regions 
1 4 1 
ajo) 1-0 On 
5-0 200 OC 
aj) [60 0-0 0-0 
2-0 0-0 Lo 
10 «(23 13 16 14 1 1 1 1 1 1 
4 3 3 3 3 
2 1 1 1 1 1 
yo |26 22 22 15 1 1 1 1,3 1.43 3 
50 |28 2 30 4 3 3 ~ 
1 1 i 1 1 
70 129 30 36 8638 4 3 3 3 3 3 
80 33 20 2 2 1 i 1 
9 25 37 33 38 y 3 3 3 3 3 
100 a ee 2 1 2 2 2 2 
vjQ)j29 4927 289 
vy(2) 30 308 
4s 2739 308 
vy 5: 38 315 
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by Thorndike (9) and Lord (5), however, this problem, for complete quotas, 
is easily solved without extensive theory by the use of subtraction. 

Consider the c;; values of Table 7 with g, = 4 and q, = 6. It is only 
necessary to form the difference c;, — c;2 for all 7 and to assign the individuals 
to job category 1 in the order of the largest value of this difference and to job 
category 2 in the order of the smallest value of this difference. The method is 
illustrated in Table 7. 

The symbol J is used to indicate the assignment so that z;,; = 1 and any 
other x;; = 0. The value of c;, is the value of c;; corresponding to x;, and 
we have 


In Table 7 this is 303 units where g, = 4 and gq, = 6 

The sum which results when assignment is made to job category 1 when 
Ci1 — Cig > 0, to job category 2 when c;. — c;, > 0, and to either job category 
1 or job category 2 when c;, — c;. = 0, is certainly an optimal one though the 
desired quotas are not necessarily met. Now since the assignments depend 
on the differences c;, — c;. , we can add any constant to c;; — ¢;2 without 
changing the resulting assignments. The value, — 3, is added to each of the 
values of c;; — c;. to form c;; — cj. — 3 in Table 7 so that the positive values 
correspond to assignments to job category 1, the negative values (including 
0) to job category 2, and the quotas are met by assigning the men in accord- 
ance with the sign of c;, — ¢;2 — 3. 

The solutions for different quotas can be obtained by forming, for different 
values of k, 


Ca — Co tk = Ca — 0) — Cee — &). (4.2) 
Assignment is then made 
to the first job category when c,;, —0 > — k, 
to the second job category when c;, — 0 < ci. — k, (4.3) 
to either job category when — 0 = cy. — 


The problem of Table 7 is worked in Table 8 by subtracting different 
values of k. The values so subtracted are placed at the bottom of the c,. 
column. Assignments are made using (4.3). The resulting assignments are 
summarized at the top of the table. The first of each pair of entries indicates 
the number of unique assignments and the second the number of ties. On 
the basis of these results a new k is selected until the quotas are met. The value 
of T is placed in the lower right-hand corner at each step. 

If the values of c;, are plotted as z’s and the values of c;, plotted as y’s, 
the line x — y + k = O divides the space into “optimal regions” with the 
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number of points in each indicating the resulting quotas. We move the line 
parallel to its original position till the desired quotas are met. These optimal 
regions for specified quotas correspond in the m-dimensional case to the 
regions bounded by hyperplanes discussed by Brogden (1) and Lord (5). 


5. The Conditions of Solution* 


For the mathematical problem of classification we need conditions of 
solution which are generalizations of (4.3). To expedite the analytic solution, 
yi; is used in place of z;; > 0 in (1.8) so that the expressions (1.9) become 


fe 
with vis = = =N. (6.1) 
Vii 


We desire to maximize 


where y/;; is 0 or a positive integer. 

An analytic derivation of the condition of solution using differential 
calculus can be obtained from an extension of the problem in which y/; is 
any real number defined by 


O< yi; < N. (5.3) 


Since c;; is also finite, J’ is bounded and there is at least one set of yi; 
resulting in the largest 7’. If these y?; are integers they constitute a solution 
of the desired problem. We use Lagrange multipliers for the side conditions 
(5.1) and have 


es + — yi) + — » (5.4) 
Taking partial derivatives with respect to each y;; we find 


_ — Ws — 


OY i; 
ay’; — u; — (5.5) 


*The initial steps in the development of this section follow the outline of a proof 
peg: by Professor D. F. Votaw, Jr., at a conference on personnel classification problems 
ye at — Research Branch, Department of the Army, Washington, D. C. in 

ugust, 1 
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Hence the necessary conditions for a maximum sum, where the maximum 
sum need not be greater than any other sum but must be at least as large, are 


Ci; — ui — = 0 (5.6) 
v;) < 0.) 


For assigned values, x;; > 0, y;; > 0 and the necessary conditions are 


= 0, (5.7) 
and in all other cases, x;; = 0, y,; = 0 and the necessary conditions are 
4 0. (5.8) 


Since there is always a maximum sum in the sense defined above (for the 
complete assignment problem) there must be at least one set of u; and v; 
satisfying (5.7) and (5.8). Hence when the sum is maximal we have 
=u; +2; for assigned values (5.9) 
Ci; S Us +9; for unassigned values. 

These conditions are used as the conditions of solution of the simplex method 
(3, 369) (11, 259). The conditions themselves are independent of the x;; which 
are positive integers (including zero) indicating the number of assignments. 

These conditions of solution are illustrated by application to the trivial 
problem of Table 1 in which f; = 1, ¢; = 1so2z,; = O or 1. The valuesv, = 0, 
v, = 2,0; = 1, uy, = 6, v2 = 6, us; = 9 satisfy the first equation of (5.9) where 
the assignments are the 13, the 21, and the 32 terms. The other values of 
C;; — U; — v; are all negative so the solution is a maximum. 

If we denote the assigned values by J, and the others by j, 11, > 0, 
2:; = 0 and 


(5.10) 


are the conditions of solution. 


6. A Generalization of Brogden’s Condition 


It seems preferable to eliminate the values of the u; from (5.10) by 
subtraction and to write the conditions of solution as 


Cis ~ 2 Vs — (6.1) 


or the equivalent 


These inequalities are the conditions of optimal solution on which the method 
of optimal regions is based. For the case of complete quotas there are always 
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maximal sums in the sense defined above and in this case the condition (6.2) 
holds. Hence there must be values of v; (and v,;) for expressing this condition 
in any numerical problem. This is consistent with the formulation of the 
problem as given by Lord (5, 300), which calls for thed etermination of a; = 0; . 
These values of v; are expressible in terms of the number system in which the 
c;; are expressed. 

Now (6.1), and hence (6.2), is in a sense a generalization of a condition 
used by Brogden in connection with his method of differences of critical 
rejection scores. He suggested that (1, 151) “Assignment is made to the first 
of two assignments when the value of the difference variable exceeds that of 
the difference score.” In the present notation we might write Brogden’s 


condition as 


where r, and r; are rejection scores with no assignment being made when 
c;; — 7; are all negative for all 7 and fixed 7. The condition (6.1), or its equiv- 
alent (6.2), might well be called Brogden’s generalized condition. 

Rao (7, 344) has proved a lemma which is very similar to (6.2). He has 
worked out an ingenious graphical-mechanical solution of the classification 
problem when m = 3 (7, 327-329). 


7. The Method of Optimal Regions 


We are now in a position to apply the method of optimal regions, which 
consists in taking values of v; , finding the corresponding quotas, and changing 
the v; until the quotas are met. The method is illustrated first in Table 9 
with the trivial problem of Table 1. 

Since one man is to be selected from each column we take the initial v; 
to be the highest c;; value in that column. Assignments using (6.2) show that, 
when the tie situation resulting from its application to the c;; values of man 
No. 3 is resolved in favor of assignment to the second job, the quotas are 
satisfied and the greatest sum is $24,000 per year. 

For some purposes we wish to know what is the maximum sum if there 
are no quota restrictions. In Table 9, for example, this amount is $25,000 
per year, which is obtained by adding the largest value in each row. The quota 
restrictions do not result here in a large percentage loss. 

Assignments made with (6.2) with v; = 0 give this completely maximal 
assignment. In the following illustrations a preliminary step to the method 
of optimal regions features the use of v; = 0. The first step then features the 
values of v; which are estimates of the v; leading to the desired quotas. 

The method is applied in Table 10 to a problem in the quota form. This 
is an abbreviation of a problem used by Brogden (1, 149). Every tenth man 
in his list was taken and the values of c;; were obtained from his values by 
multiplying by 10 and adding 25. The quotas adopted for this problem are 


et 
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q = 4,@ = 1, ¢; = 4, qs = 1. The greatest possible T, without regard to 
quotas, is 344 units. 

The initial values of v; , the v;”’, are determined by selecting the fourth 
largest c;, , the largest c;. , the fourth largest c;; , and the largest c;, . The 
tentative assignments are made using (6.2) and the resulting quotas are 
indicated in rows above the table. The first entry is the number of unique 
assignments and the second is the number of ties. 

The resulting g;”’ are compared with the desired gq; . The initial assign- 
ment gives an excess of 2 in the first job category, a deficiency of 1 in the 
second, and a deficiency of 1 in the fourth. An excess indicates the value of 
v; is too low and a deficiency indicates the value of v; is too high. The value 


of v{” is raised by one unit to form v;” and the values of v;” and v{” are 


lowered by one unit to form v3” and v{”. The values of J® and q‘” are 
computed. Values of v{* are then assigned. After four steps with the assign- 
ment of ties, the quotas are met. 

The method is called the method of optimal regions, since the changing 
of the values of v; may be interpreted as the moving of hyperplanes parallel 
to original positions in such a way that the optimal solutions involving the 
number of points within the resulting regions (or on the boundaries) even- 
tually satisfy the desired quotas. 

The method of optimal regions is based on the fact that bounded values 
of v; exist which identify the regions associated with the desired quotas. A 
primitive form of the method calls for the calculation of the quotas of each 
different set of v; values (or perhaps the calculation of those sets close to an 
approximate set such as v;"’). This form of the method is not practical except 
in trivial problems. A less formal variation of the method, suitable for 
problems of hand calculation but not of machine calculation, calls for estimates 
of the values of the increments to the vS”’ in accordance with the values of 
q‘ — q; . There is no guarantee, of course, unless an appropriate estimation 
scheme is specified, that such estimates will lead to an improvement at any 
specific step, but in many hand computation problems the alert computer 
can arrive at the result in relatively few steps. Larger numbers of job cat- 
egories, larger numbers of rows, and large positive correlations between the 
columns of c;; values tend to increase the number of steps. 

A more formal variation of the method, and one which is better adapted 
to machine computation, calls for the calculation, and ordering for fixed 7 
and k, of the values of c;; — vs"? — c;, + 5°. These values are all non-positive 
as indicated by (6.2). This detailed method of optimal regions calls for the 
determination of the increments to the v{“ by selecting g; — q‘*’ largest 
values of c;; — v{ — c,, + v¥? in the same way that the values of v‘"’ are 
selected using the largest g; values of c;; . There is no absolute guarantee that 
the process will converge to a solution, though the method of exhaustion 
does guarantee a solution after a finite number of steps. The method is simply 
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an iterative process in which, at each step, we see where we are and where 
we want to go. We take steps, which for the most part can be mechanized, 
toward the objective. 

In practice there seems to be little difficulty in obtaining a solution in 
a relatively small number of steps except when the objective is not clear 
because of large numbers of ties. These make it difficult to discover if the 
q; can be obtained from the q}”. This is precisely the problem faced by Smith 
(8, 45) with the method of bounding sets. A solution has been given by Votaw 
and Dailey (12, 16) which is practical, for hand computation, when the 
number of job categories is less than eight, and for larger values with machine 
computation. Variations in the method of interchange (4) may also be used 
in solving this problem, which has been called “the quota problem” by Votaw 
and Dailey (12, 15). 

Because of the multiplicity of ties in rows having high frequencies, the 
quota problem is commonly more serious in connection with problems 
naturally expressed in the frequency form (transportation problems) than 
those naturally expressed in the quota form (personnel classification problems). 
As an illustration involving multiple ties the method of optimal regions is 
applied to a problem naturally expressed in the frequency form in Table 11. 
This is essentially a problem used by Votaw (3, 259) to illustrate the simplex 
method. The form of Table 11 presents both the statement of the problem 
and its solution. The values of v$’ are determined by finding the 35th of the 
C;, values, the 35th of the c;. values, and the 30th of the c;,; values, considering 
the frequencies and measuring from the largest values of ¢;; . 

Ties, many of them, result in the first and fourth man categories. Can 
these ties be resolved so as to satisfy the quotas? A necessary condition is 


TABLE 12 TABLE 12 
Frequency Form with Optimal Regions Quota Form with Rejection Scores 
20-60 20-20 o-L0 3-0 0-0 240 10 
a, | 35 35 | gC) 2-0 0-0 
0 0 Blo k35) (35) (30) 
B12 8 5 
100 30 25 24 241,2] 1] 2 
rj} 0 0 0 0 [236 
rj} 30 33 189 
29 ad 
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that the available values of g"’, including ties, must be greater than q; for 
each value of j7. We see at once from the upper rows that 


g. = 20 + 60 > gq, = 35; g2 = 20 + 20 > q. = 35; 
gs = 0+ 40 > gq; = 30. 


If this necessary condition were not met, we would take values of v{” in the 
direction of meeting it. Here we apply the additional necessary conditions 
involving combinations of job categories: 


9i2 = 80 > = 70; gis = 80 > = 65; 923 = 80 > = 65; 
Ji23 = 100 > Qi23 = 100. 


The satisfaction of all these necessary conditions constitutes a sufficient 
condition and the quotas can be met. The resulting values of z;; are shown 
at the right of Table 11. The value of T is 825 units and the quota restrictions 
cause a loss of but 15 units. 

Frequently, as in this case, the method of optimal regions gives a very 
satisfactory solution of problems naturally expressed in the frequency form, 
even though the matter of ties sometimes demands close examination. The 
method is recommended, however, primarily for problems which are naturally 
expressed in a quota form featuring a relatively small m. 

The method is applicable to non-trivial problems. A classification problem 
with 1152 man and 7 job categories, and highly correlated columns of ¢;, , 
was worked with it in eight iterations. 


8. The Case of Incomplete Quotas 


A problem involving incomplete quotas can be worked with the method 
of optimal regions after the problem has been transformed to a problem with 
complete quotas by the use of the device illustrated in Table 6. However, 
the method of critical rejection scores may be used instead. This method is 
especially effective when the quotas are (relatively) small numbers. The 
method of rejection scores is a slight modification of the method of differences 
of critical rejection scores introduced by Brogden (1). This modification 
uses a method of solution very similar in form to that of the method of optimal 
regions described above. 

Brogden used rejection scores in indicating a preliminary assignment. If 
r; is a rejection score, assignments are tentatively made to job category 7 only 
if 

> 0. (8.1) 


The use of (8.1) may lead to the tentative assignment to more than one job 
category. Conflicting assignments are then resolved by assigning the individual 


a 
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to job category J in accordance with Brogden’s condition (6.3). This may be 
written in the alternative form 


It is not necessary to use (8.1) and then Brogden’s condition. The same result 
can be obtained with the use of (8.2) and the additional specification c;; — 
r, > 0. If this is not satisfied for every value 7 = J, no assignment is made. 

The technique of the method follows that of the method of optimal 
regions with r, playing the role of v, , r; of v; , except that no assignment is 
made to individual 7 when every c;; — 7; < 0. 

The method is illustrated in Table 12 where the problem of Table 10 
with gq, = 2, g@ = 1, qs = 2, qs = 1 is solved with the method of rejection 
scores. The values r; = 0 are first used. The six largest c;; resulting are added 
to obtain the maximum sum disregarding quotas. 

The method of rejection scores can be used to solve the complete assign- 
ment problem. In this case one takes successively smaller values of the r; 
until the quotas are met. 


9. Concluding Remarks 


The personnel classification problem, which is mathematically the 
equivalent of several other problems in the social and biological sciences, is 
really a special problem in linear programming. Though the techniques, 
such as the simplex method, of the general linear programming problem are 
available. it seems preferable to direct the mathematical solution toward the 
simpler mathematical model of the classification problem. When this is done 
a condition for assignment, which is a generalization of Brogden’s condition, 
is available and serves as the foundation of the method of optimal regions. 
This method, which is based on concepts related to those introduced by 
personnel workers such as Brogden, Thorndike, and Lord, seems particularly 
effective when applied to the personnel classification problem. 
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Addition at proof reading: The detailed method of optimal regions has been further improved 
with the introduction of devices for automatic determination of the successive sets of 
u; which converge to a solution. Preliminary transformations lead to initial values of v; 
which, in all the problems worked thus far, become solutions with two or three iterations. 
For example, the 1152 X 7 problem mentioned above has been solved by determinin 
sets of v; from a preliminary transformation followed by two steps of the detailed meth 


of optimal regions. 
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It is first demonstrated that Aitken’s selection formulas are equivalent 
to a linear transformation in the factor space. On this basis the Thomson- 
Ledermann theorem concerning the invariance of the number of common 
factors under selection, and a theorem concerning the invariance of factor 
loadings under selection are derived. A mathematical proof of the results of 
Thurstone, which are concerned with the invariance of simple structure 
under selection, is given. The paper provides a conclusive answer to the 
question, considered by Thurstone and Thomson, whether a multivariate 
selection is always reducible to successive univariate selections. 


I. Selection as a Linear Transformation 


The effect of a selection of population (i.e., selection of subjects in 
psychological experiments) on correlations and variances is indicated by 
Aitken’s formulas 

Vie Vi = Vis (1) 


(see 1, 2, 3 or 4), involving the inverse Rj} . As is well known, the inverse of a 
square matrix exists only if the matrix is nonsingular, i.e., if its determinant 
does not vanish. In virtue of this, we must have | 2;; | # 0. On the other 
hand, it follows from this condition that the rows and the columns of R,,; 
must be linearly independent. Accordingly, we can state the following 
theorem: 


Se.ection THeoreM I: Selection tests are linearly independent in a 
selection determined by Aitken’s formulas. 


We then pass over to a geometrical mode of representation customary in 
factor analysis and let each test be represented by a vector of unit length. 
We call the manifold spanned by the | “selection test vectors” (i.e., vectors 
that geometrically represent the selection tests) the ‘“‘selection space.” Since, 
in virtue of Selection Theorem I, the / selection test vectors are linearly in- 
dependent, the selection space is /-dimensional. It follows directly from the 
definition of the selection space that selection test vectors are entirely in 
this space. Other test vectors, on the other hand, may have—in addition to a 
selection space component—another component outside of this space. We 
then set up an arbitrary coordinate system in the selection space and express 
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the projections of the selection test vectors on the coordinate axes by the 
matrix || a;, || = A; , where the representative element a;,, expresses the 
projection of jth selection test vector on the-mth coordinate axis. On the 
other hand, let the. matrix || a., || = A, express the projections of the non- 
selection ‘test vectors on the coordinate axes of the selection space, the 
representative element a,,, expressing the projection of the kth non-selection 
test vector on the mth coordinate axis. The scalar products of selection test 
vectors and the selection space components of non-selection test vectors 


then have the form 
Pis = AjmAsm 


(2) 


where j and J refer to selection tests and k and K to non-selection tests. As 
the selection test vectors are entirely in the selection space, their inter- 
correlations are simply 

Tis = Piss 

Tik = Piz 
On the other hand, as the non-selection test vectors are not —_—es 
entirely in the selection space, we have 


Tex = Pex + Pex, 


where );x is that part of the scalar product of the test vectors k and K which 
originates outside of the selection space. In matrix notation, we can express 
the foregoing as 


= A;Aj, (3) 
Ri A;Aj (4) 
Ru = A,Ai + Ru. (5) 


In the selection space we then perform a linear transformation, converting 
the selection test vectors 7 into new vectors whose projections on the co- 
ordinate axes are b;, . Transformation coefficients l,, , [° in number, are 


determined by linear equations, 


(j = 1,2,---,D 
(p = 1, 2, 


The new coordinates of the non-selection test vectors can then be. ered 
from equations ; 
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bin = « (k= 
(p = 1,2,--- ,D 
The above sets of equations can be written in matrix notation as 


B= Ad, (6) 
B, = A,L, 

the matrix of transformation being 
L = A;'B,. (8) 


The scalar products of the new, transformed test vectors in the selection 
space are 


wis =X 

= 
and the new covariances of the tests accordingly 

= Biss 

= Bik» 


Ux = Mex + Pax - 
In matrix notation: 


Vis B;B} ’ (9) 
Vie = BB, (10) 
1 \ Vex = t+ Ru . (11) 


From (10), together with (7), (8), (4), (9), and (3), we have successively 
Vin = Ay = 
= V,(A,;A) = (12) 
and from (11), together with (7), (5), (4), and (3), we obtain — 
Vis = + Ru — 
= Ru + 
= Ru — + Ry 
= Ru — — - 
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In point of fact, however, (12) and (13) are simply Aitken’s formulas 
(1), so that we have proved the following: 


SELECTION THEOREM II: Selection, as regards its effects on correlations 
and variances, is equivalent to a linear transformation in the selection space. 


As far as a univariate selection is concerned, the matrix of transforma- 
tion, L, is reduced to a single element, viz., to the new standard deviation of 
the single selection test. Therefore, univariate selection simply corresponds 
to a “contraction” or “dilatation” of the test vector space in the direction 
of the selection test vector (see 1, 444). The question of whether a multi- 
variate selection can always be reduced to successive univariate selections 
has been considered by Thurstone and Thomson (see 1 and 2). This question 
is answered by our theory in the affirmative. For a linear transformation in 
an [-dimensional space is determined by I? transformation coefficients, and 
it is always reducible to | successive “contractions” and “dilatations” taking 
place in various directions of this space. A single contraction or dilatation 
of this kind in a given direction is uniquely determined by / — 1 ratios of the 
direction cosines together with one contraction or dilatation coefficient; 
i — 1 + 1 = 1 conditions can accordingly be imposed upon any univariate 
selection, and consequently I? conditions upon / successive univariate selec- 
tions. On the other hand, these | successive univariate selections obviously 
define a given linear transformation in the selection space. By letting this 
linear transformation satisfy the I’ conditions that its / transformation 
" coefficients are I, , i.e., the same as those of the linear transformation cor- 


responding to a given multivariate selection, we can reduce a given multi- 


variate selection into / successive univariate selections. 


SeLectTion THEOREM III: A given I-fold multivariate selection is always - 


reducible to | successive univariate selections. 


II. On the Invariances in Selection 


We now substitute communalities for the unities in the principal diagonals 
of R;, and R,, to obtain the reduced correlation matrices Rf and Rf . For 
the sake of uniformity we also shall subsequently denote the matrices R;, and 
R,; by R&A and R} , respectively. Instead of the complete correlation matrix 
R we now have the reduced correlation matrix R*. Let us assume that by 
factoring this matrix we obtain r orthogonal common factors. We partition 
the resulting common féctor matrix F of order n X r into two submatrices, 
viz., into an 1 X r submatrix F; , comprising the selection tests, and an 
(n — l) X r submatrix F, , comprising the non-Sttection tests (see Table 1). 
In virtue of the basic theorem of factor analysis we then obtain 
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R* = FF’, 
Ri = F,F;, 
Ri = = Ri,, 
Ri = (14) 


In Table 1 we write, besides the common-factor matrices F; and F, , 
the unique-factor matrix of the selection tests, U; , which is, as a matter of 
course, a diagonal matrix, and a matrix of the non-selection tests in terms of 
the unique factors of the selection tests, which is, as a matter of course, a 
null matrix: We denote the total matrix thus constructed Fy . 


r l 
nll 10 n-l| F, 
Y, r 1] FL+UL, 
nl| F |o! & AL. 


It was shown in the first section of this paper that selection is equivalent 
to a linear transformation in the selection space spanned by the selection 
test vectors. Since this /-dimensional selection space is evidently a subspace 
of the (r + l)-dimensional space comprising the r-dimensional common- 
factor space and the /-dimensional unique-factor space of the selection tests, 
a linear transformation, L, which we perform in the selection space, at the 
same time means a given linear transformation L, in the above-mentioned 
total space. For this reason the factor matrix F, is transformed in selection 
into the matrix FL, (see Table 2), giving the new after-selection covariances 
and common-factor variances according to the formula 


V* = (FoLo)(F ole)’. (15) 


| 
j 
j 
TABLE 1 TABLE 2 
\ 
\ j 
\ 
j 
‘ 
i 
4 


‘ 
= e 
= 


32 PSYCHOMETRIKA 


We partition the matrix L, of order (r + 1) X (r + JD in the manner 
indicated by Table 2 to get an r X (r + 1) submatrix L- , corresponding to 
the transformation of the common-factor space, and an 1. X (r + 1) sub- 
matrix Ly corresponding to the transformation of the unique-factor space 
of the selection tests. Employing this notation, F;L¢ + U;,Ly is that sub- 
matrix of FL which includes the selection tests, and F,L¢ that submatrix 
of F,L,) which comprises the other tests (see Table 2). Equation (15) then 
yields 

Vi = (PiLe + UjLy)(FiLe + 


Va = + U;Ly)(File)’ = Vi; , (16) 
Vs = (F zlic)’. 


The new, after-selection correlations and communalities are obtainable from 
the new, after-selection covariances and common-factor variances, represented 
by (15) and (16), in accordance with the formulas 


= DV*D = (DF oLa)(DF ole)’, 

= D VAD, = (DP Le + Le + D,U;Ly)’, 

Ri = VAD, = (DF Lc)(DiF Le)’ + Ly)(DiF Le)’, 

Ri = DiVAD. = (DF Le)(DiF Lc)’; (17) 


where D, D; , and D, are diagonal matrices, the reciprocal values of the new, 
after-selection standard deviations of the tests being their elements. We see 
that the new, after-selection common-factor matrix is not DFL- , which is 
obtained from the matrices D,;F;Lc¢ and D,F,L- , but the new common- 
factor matrix js DF,L, . The rank of DF,L, accordingly indicates the number 
_of common factors which are obtained after selection. Let us determine this 
rank. 

We know, for one thing, that the rank of a product matrix cannot be 
higher than that of any factor. Consequently, the rank of DF,L» is at most 
equal to the rank of Fy . On the other hand, the diagonal matrix D always 
has an inverse, and the square matrix L, is in general non-singular; hence 
we have, in the general case, 


= Fo. (18) 


From (18) it is evident that the rank of DF,L, cannot be lower than the rank 
of F,, so that, in the general case, the rank of DF,L, is the same as the rank 
of F, . As the matrix F, in Table 1 has r + 1 linearly independent columns 
and n rows, the rank of Fy is equal to the smaller of the numbers n and 
r + l. As this is also the rank of DF,L, , and as the inequalities n > r and 
tr +1> r always hold, the number of common factors has been increased by 
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selection, but it cannot exceed the number of the original common factors 
by more than I. We then show that the additional factors appear only in the 
selection tests so that the number of common factors of the non-selection 
tests is not increased. The new, after-selection common-factor matrix of the 
non-selection tests is D,F,L¢ , as is evident from (17) and from what was 
said above. As a product rhatrix cannot have a higher rank than its factors, 
the rank of D,F,L¢ cannot be higher than the rank of F, . But this is to say 
that the number of common factors of the non-selection tests is not increased 
by the selection, and that the additional factors appear only in the selection 
tests. And it follows from this that in univariate selection the single additional 
factor, appearing in the single selection test, can be interpreted as a unique 
factor of this test. Thus it is possible, in univariate selection, to keep the 
number of common factors invariant as regards the selection test also. To 
sum up, we come to the Thomson-Ledermann theorem (see 5, 6, and 7): 


INVARIANCE THEOREM I: The number of wmmon factors is not increased 
by selection except in the selection tests, in which there appear at most as many 
additional factors as there are selection tests. In univariate selection the number 
of common factors is not increased as regards the selection tests either. 


Our next concern will be the invariances appearing in the factor loadings 
uader selection. As our intention is not to confine ourselves merely to the 
case of orthogonal factors, we first rotate the common-factor matrix F into 
an arbitrary common-factor matrix in the following manner. We supplement 
the factor matrix F, in the way shown in Table 3, by an arbitrary matrix of 
order r X r, and normalize the rows of the last mentioned matrix. We thus 
obtain the matrix 7. After having made sure that the rows and columns of 


T are lincarly independent, we can regard the elements of 7’ as the loadings © 
of some rotated factors in terms of the original orthogonal factors. Correla- | 


tions between the tests and the rotated factors are, in virtue of the basic 
theorem of factor analysis, 


S = FT’. (19) 


The loadings of the rotated factor matrix S, thus obtained, do not remain 
invariant, but are changed by selection in the same manner as are other 
correlations. But we need not express the loadings of the tests in terms of the 
rotated factors as the correlations S. We can express the tests also as linear 
combinations of the factors by writing 


F = AT (20) 
(see 1, 353) and solving for the unknown A: 
A = FT". (21) 
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Finally, we partition the matrix A into submatrices A, and A, in accordance 
with the equations: 
F, = A,T, (22) 


F, = A,T, (23) 


and show that the factor loadings of A, display a given invariance in the 
selection of population. 

As already mentioned, D,F,Lc¢ is the new, after-selection common-factor 
matrix of the non-selection tests, its elements being accordingly the after- 
selection loadings of the non-selection tests in terms of orthogonal factors. 
Analogously, it is evident (see Tables 2 and 3) that the elements of D,TL- 
are new, after-selection loadings of the—not necessarily orthogonal—rotated 
factors in terms of the orthogonal factors, D, being a diagonal matrix normal- 
izing the rows of TL; . As a matter of notation we put 


F, = DF Le , (24) 
T= D,TLe. (25) 


It follows from (25) that the rank of ,7 cannot be higher than the rank of 
L- . On the other hand, a diagonal matrix always has an inverse, and as we 
_ had made sure that the rows and columns of 7 were linearly independent, 
T also has an inverse. Thus we can set up the equality 


= Le (26) 


from which it is evident that the rank of ,7' cannot be lower than the rank of 
Le either. Hence, the rank of ,T is the same as the rank of Le . As Lg is, 
according to Table 2, of order r X (r + J), its rank is, in the general case, 
r. Consequently, the rank of ,7 is also r in the general case so that, in the 
general case, the rows ,T' are linearly independent. But this is to say that, in 
the general case, we have the equality ~ 


= A, (27) 


and that we are able to express the non-selection tests after selection also as 
linear combinations of the rotated factors. ,A, in (27), which has so far been 
unknown, can be expressed in terms of A, , i.e., the matrix corresponding to 
it prior to the selection, in the following way: From (23) and (24) we have 
first 


= D,A.,TLe = D,A,D7;'D,TLe (28) 
From this it follows, in virtue of (25), 
oy = (29) 


Comparing (27) and (29) we find 
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= D,A,D;". (30) 


As both /, and D, , and hence also D7’, are diagonal matrices, it is evident 
from (30) that the new, after-selection factor matrix ,A, is obtainable from 
the original factor matrix A, by multiplying the rows and columns of the 
latter by given numbers. A simpler kind of invariance of the factor loadings 
is encourtered if it is assumed that the after-selection factor analysis is 
carried out with the covariances rather than the correlations as the starting 
point. According to (16) FiLc is the new, after-selection common-factor 
matrix of the non-selection tests, provided that this after-selection analysis 
has been based on covariances. On the other hand, premultiplying (27) by 
D,', we obtain 

Dy'.F, = Ax .T, (31) 


which gives, in virtue of (24) and (30), 
= (A,D;').7. (32) 


Consequently, A,D;' is now the new, after-selection factor matrix corre- 
sponding to the above .A, . Since D;' was a diagonal matrix, A,D;', is 
obtainab|: from the former factor matrix, A, , by multiplying the columns of 
the latter by given numbers. In other words, the columns of the after-selection 
factor m: ‘rix A,D7" are proportional to the columns of the original factor 
matrix A, . Thus we have proved the following: 


INVARIANCE THEOREM II: When expressing the tests after selection as 
linear combinations of the same common factors as before selection, the columns 
of the new factor matrix are proportional to the columns of the original factor 
matrix arrived at in the same way, provided that (1) the after-selection factor 
analysis has been carried out on the basis of covariances; and that (2) the selection 
tests have been excluded. 


We then assume that the factor matrix Asi is one representing a “simple 
structure.”’ This is to say (1, 335) that (1) every row of A, contains at least 
one zero, (2) every column of A, contains at least r zeros for tests that are 
linearly independent, and (3) for each pair of columns there are several tests 
whose element in the one column is zero, but‘in the other column nonzero. 

It was stated above that the new, after-selection factor matrix ,A, is 
obtainable from the original matrix A, , in virtue of (30), by multiplying the 
rows and the columns of the latter by given numbers. Hence, the zero loadings 
are exactly in the same places in ,A, as in A, . From this and the above 
definition of a simple structure there follows directly: 


INVARIANCE THEOREM III: If that part of a test battery which is let when 
the selection tests are excluded reveals a simple structure, this simple structure 
is invariant under selection in the general case. 
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Selection tests have been excluded from Invariance Theorems II and 
IIT, since these tests do not reveal the kind of invariance with which these 
theorems are concerned. The task of proving that this is the case, we leave, 
however, to the reader and will confine ourselves to another circumstance 
connected with these two invariance theorems. 

Our Invariance Theorem III is stated with the proviso that the invariance 
concerned appears only in the “general case.’’ We had to make this reserva- 
tion owing to the fact. that this theorem was based on equation (30), which, 
in turn, was a consequence of (27). As was emphasized when the equality 
. (27) was established, it is valid only in the general case when the rank of the 
matrix of transformation L- , and consequently also the rank of ,7’,, is r. In 
the exceptional case when this rank is less than r, there is linear dependence 
among the rows of ,7’. As the T-factors have been rendered linearly dependent 
in selection, it is not possible any more to express the tests, after selection, 
as linear combinations in the manner presupposed in (27). 

We shall show now that this exceptional case, when the rank of ,7 and 
consequently also the rank of the transformation tatrix L¢ is less than r, 
may happen if the selection is “total” in some of the tests. Selection is called 
total in test j if the new, after-selection variance of j, 8; , vanishes (see 1, 
443). On the other hand, of course, this new variance, s; , is the jth element 
of the principal diagonal of the matrix V;; = V#. If we put F; = || cj» ||, 
U; = || u;; ||, Le = || lap ||, and Ly = || 1,, ||, we have from (16) the condition 
of total selection in the form 


Provided that u;; = 0, this is reduced to 
Calne)” = 0 (34) 
or 
Dd cinlnn =O (P= 1,2,---,D. (35) 


The equations (35) give expression to linear dependence among the rows of 
Le = || lap ||, in which case the rank of Le is reduced and, consequently, less 
than r. On the other hand the condition u;; = 0 means that the communality 
of the test j is unity. Thus, we have proved: 


INVARIANCE THEOREM IV: If the selection is total in a test whose com- 
munality is unity, simple structure is not invariant under selection, and the 
tests can no longer be expressed as linear combinations of the same factors as 


\ 


before selection. \ 

Thurstone (1, 440-472) has not paid attention to the condition u;; = 0, 

being of the opinion that a simple structure as a whole will never be invariant 
under total selection. 
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III. Numerical Example 


Thomson and Ledermann (5, 6, and 7) have given numerical examples 
illuminating our Invariance Theorem I. Thurstone again has provided 
examples for our Invariance Theorems III and IV (see 1, 440-472). In order 
to illustrate our Invariance Theorem II as well, we make use of a numerical 
example constructed by Thurstone. 

When factoring a given correlation matrix, the matrix F shown in 
Table 4 was obtained by Thurstone. We now form the matrix 7 by normal- 
izing the three first rows of F. We use the pivotal condensation method to 
find the inverse of 7’, which gives us T~' in Table 5. From Tables 4 and 5 we 


TABLE be TABLE 6 
I Ir III A B c 
1 33h 688 TABLE 1 837 000 » 
2 63h  ob8 2 015 837 
5u3 3 000 837 
638 379 385 430 6L7 439 72h 920 
5 759 5 Ws o7 
6 637 830 O61 +533 6 
7 561 359 7 
8 -387 8 oi9 «6418 725 
9 196 9 
lo 820-075 151 10 509-83 505 
*Decimals omitted in all 
tables 
TABLE 7 TABLE 8 TABLE 9 
I II III A B c A B c 
1 265 =-270 1 1 
2 2 O1h 833 003 2 833 003 
3 73 3 000 836 2 Olh 836 
620 33 4 343. «760 330-730 
5 160555 5 657 88 020 5 O17 
6 620 =519 ~-165 6 003s 77h 6 328 
7 566 257 7 61 7 S79 007 
8 70 O51 = 8 76s 8 023) 
9 70-336 =195 9 019 «722 9 019 722 
10 815 -02h 10 399 530 10 378 BS SO 


can calculate the matrix A = FT™', shown in Table 6 [cf. equation (21) 
above]. We then select a new population so that the standard deviation of 
test 1 is changed from unity to .60. The intercorrelations of the tests in this 
new population have been computed by Thurstone according to Aitken’s 
formulas. In addition he has also carried out a factor analysis on the basis of 
this new, after-selection correlation matrix and obtained the new factor 
matrix ,F shown in Table 7. By normalizing the three first rows of ,F we 
obtain a factor matrix ,7’, and it turns out in this case that ,7' is identical 
with T. We therefore obtain the matrix ,A, in (27) as a product, ,F,7T™’, 
which can be computed from Tables 7 and 5. The new factor matrix ,A, 
which is attained is shown in Table 8. To perform the after-selection factor 
analysis on the basis of covariances instead of correlations, we multiply the 
rows of ,A, by the new standard deviations of the tests. This multiplication 
yields the matrix A,D;* appearing in (32). As the new standard deviations 
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of the tests 2, 3, ---, 10 are 1.00, 1.00, .960, .875, .960, .875, 1.00, 1.00, and .946, 
the resulting matrix A,D;" is that shown in Table 9. According to our theory 
the columns of the matrix of Table 9 and the matrix of Table 6 should be 
proportional, provided that selection test 1 is excluded. As a matter of fact, 
this is the case, for the matrix of Table 9 is obtained from that of Table 6 
by multiplying the columns of the latter by 1.315, 1.00, and 1.00, respectively. 


4. 
6. 
7. 
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The square root method for the solution of a set of simultaneous linear 
equations or the. reduction of a matrix has been known for some time under 
a variety of names. Because of its usefulness in statistical work, especially in 
factor analysis, the square root method is presented in general terms and an 

. example given. Several independently developed ‘multiple group methods” 
for factor analysis are compared and synthesized. Their fundamental concepts - 
are set forth and an appropriate system of notation developed. Detailed 
computational procedures are outlined, and the square root method is 
emphasized as a computing aid in multiple group analysis. 


There are two principal objectives of this paper: (1) a clarification and. 
coordination of several approaches to multiple group methods of factor 
analysis; and (2) the application of the square root method to the multiple 
group method of factoring a correlation matrix. In the process of meeting 
these objectives, a brief historical sketch of these techniques will be presented 
and the formal procedures will be developed and illustrated. 


1. Historical Note 


While no attempt is made to give a complete and exhaustive account of 
the history of the multiple group methods of factor analysis, a short account 
of the highlights in this development seems to be in order. Recently, Guttman 
(8) called attention to the basic werk he had done in multiple group methods 
of factor analysis, which work apparently was overlooked by Thurstone (18) 
in his account of the similarity between the “simple method of factor analysis” 
proposed by Holzinger (11) and the “multiple group method of factoring 
the correlation matrix”? which he proposed (16). A search of the literature 


discloses definite evidence of work in this direction prior to any of these 


papers. 

It matters little “who got there first,” but there are indications that in 
1937 Horst (13) anticipated the multiple group method of factor analysis. 
His was a theoretical presentation, however, and the lack of computational 
procedures apparently was the reason that the method was not adopted and 
developed. Several other writers dealt with group factors in the early 1930's. 
Notable among these were Cyril Burt, who considered the “group factor 


*The opinions expressed in this paper are those of the author and do not necessarily 
reflect official Department of the Army policy. The author is now with The RAND Corp- 
oration, Santa Monica, Calif. 

~ 


89 


4 
' 
ai 
: 
: 
7) 
| 
i 
| 
| 
| 
| 
at 


40 PSYCHOMETRIKA 


_ method’’; R. C. Tryon, who proposed “cluster analysis; and K. J. Holzniger, 

who developed the “bi-factor’” method of analysis. While these methods 
certainly involve the group factor concept, they are not specifically in the 
spirit of the multiple group methods of factor analysis under consideration 
in the'present paper. Guttman’s presentation (7) of the theory in 1944 met 
with the same fate as Horst’s paper, and for the same reasons. However, 
when Holzinger (11), in the same year, and Thurstone (16), a year later, 
" presented simple computing procedures for “group factor analysis,” there 
was ready acceptance, even though the similarity of methods was not recog- 
nized for several years (17, p. 171, 18, 8). 

The history of the development and use of the “square root method” 
is even more vague. Certainly, as a formal mathematical procedure for the 
solution of a set of simultaneous linear equations or the reduction of a matrix, 
it must have been discovered over and over again, and may go back to the 
time of Gauss. Perhaps tne earliest application of a square root method to 
the solution of normal equations in least squares theory was made by Com- 
mandant A. L. Cholesky of the French Navy around 1915, and published 
after his death by Commandant Benoit (3) in 1924. It was rediscovered by 
Banachiewicz (1, 2) in 1938 and presented as an efficient means for solving a 
system of linear equations and for the calculation of determinants and their 
inverses. The square root method was introduced in the American statistical 
literature in 1944 by Dwyer (4), who emphasized its use in correlation and 
regression (5) and who showed the relationship of this method to other 
methods of linear computation (6). 

Concurrent with this development of the square root method as a means 
of solving formal mathematical and statistical problems, essentially the 
same technique was being devised specifically for factor analysis. It was 
recognized that a factor analysis of a set of variables whose intercorrelations 
constituted a symmetric matrix could always be obtained by a general 
algebraic procedure known as “completing the square.’”’ The method was 
applied specifically to a correlation matrix by McMahon (14) prior to 1923. 
Then during the rapid development of factor analysis theory in the 1930’s, 
it was independently developed as the “diagonal method” by Thurstone 
(15, p. 78) and as the “solid staircase method” by Holzinger (10). Since these 
methods were designed expressly for factor analysis, they did not present 
general computing techniques. Nevertheless, they are special instances of the 
square root method, with the broader implication of a technique for linear 


computations in general. 


\ 


2. The Square Root Method 


While the square root method has been featured in several papers and 
applied to a variety of statistical problems during the past decade, it still is 
relatively unfamiliar to many researchers who use the Doolittle worksheets, 
As pointed out by Dwyer (6, p. 115), the advantages of the square root 
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method over the Gauss-Doolittle method are that it is more compact, re- 
quiring less recording, and. that it permits greater ease in finding the entries 
to be used. Not only is the square root method more expedient for solving a 
symmetric set of equations, but it is especially useful in obtaining the inverse 
matrix in solving problems in statistics. 

For the foregoing reasons,'the square root method will be presented here 
in some detail. The procedure will be outlined in general terms and illustrated 
with a simple problem (9) involving the least-squares prediction of a de- 
pendent variable from three independent variables, viz., 


= Bi2, + Bote + , (1) 


where 8; is used for the conventional @,;..2), in which the number in paren- 
theses merely shows how many variables are held fixed. The normal equations 
in this case are: 


+ 128s + 1138s = 
T2181 + + T2383 = To (2) 
7318, + T3282 + 
where, of course, the conditions for symmetry r;; = 1;; are satisfied and 
r;, = 1 fort,j = 1, 2, 3. 

_ The computing procedure for the determination of the regression co- 
efficients in equation (1) is indicated in general terms in Table 1, and illus- 
trated with specitic numerical data. The step-by-step procedure, immediately 
following, is readily extended to any number of variables. 

Step 1. Enter the intercorrelations among the independent variables 


and their correlations with the dependent variable on the first 
three lines of the Work Sheet. 


Step 2. Obtain the sums by rows, i.e., 


t; = = 1,2, 3) 


" Note: Entries in Check column for Lines 1, 2, 3 are described in 
Step 10. 


Step 3. The first step of the square root method is now applied by 
using 7), as a pivot. The first element in Line 4 is given by 


81 = 


while the remaining elements are obtained by the formula: 


= (i > 1) 


Note: Since r,, = 1, the elements of Line 4 are cq respec- 
tively, to the elements of Line 1. 
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TABLE 1 
The Square Root Method 
Independent Variables | Dependent 
Variable 
Line Total Check 
24 
General Solution 
S14 Sit Sit 
6 $3412 S3t-12 
1 R 
By Bo B3 4°123 4°123 
Numerical Dlustration 
1 /1.000 .693 .21@ 571 2, 480 571 
2 * 1,000 .295] .691 2. 679 691 
3 * *  1,000| .456 1.967 456 
4 11,000 .216} 2. 480 2. 480 
5 410 1, 332 1. 333 
6 95 . 262 1.217 1.217 
ard . 563 750 


*Terms below the diagonal of a symmetric matrix are deleted for 
simplicity. Terms below the diagonal of the "square root" matrix 
are actually zero, and are simply omitted. 


| 
. . 
\ 
\ 
i 
. 


. 
wa 
| 
| 
| 
| 
| 


Step 4. 


Step 5. 


Step 6. 


Step 7. 


Step 8. 
Step 9. 


Step 10. 


Step 11. 
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The calculation in the Total column of Line 4 is carried out as 
for any other column, yielding s,, . This value should agree, 
except for rounding errors, with the sum sj, (Check column) 
of all elements computed in Step 3. 

The formulas for the elements of Line 5 are: 


t 
890-5 = — 


= Tas (i > 2) 
822-1 


Check Line 5 by comparing the calculated value, on , with 
the row sum, 83,.; . 
The formulas for the elements of Line 6 are: | 


833.42) = VTs3 — 813 — 823-1 


— 81813 — 3) 


§3i-(2) = 


where the notation s;;.:2) is used instead of the specific 83;.12 
to ‘suggest an easy generalization when the number of variables 
already eliminated is more than 2. . 

Apply row sum check to Line 6. 

The values of the regression coefficients are obtained by appli- 
cation of the following formulas (back solution): 


833-12". 
8241 — 823-183 
2 : ’ 
82201 

814 — 81383 — 81282 
$11 


= 


A check on the entire computations can be made by substituting 
the regression coefficients back into the normal equations (2). 
The results are designated by r{, , Ts , 73, and should agree 
(except for rounding errors) with the original correlations of 
independent with dependent variables. 

The multiple correlation coefficient can be computed by use 
of the usual formula involving the 6’s and r’s, viz., 


= + Botan + Bata - 
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From the formal solution of the three-variable problem it can be verified 
that 


Tia Ma ‘181 811 832 813 
: To. Tes | =| 812 8221 O O Bea |e (3) 
T31 Ts3 8:13 823-1 833-12 |LO 0 833 +12 


More generally, the square root method can be formulated in matrix notation, 


as follows: 
R= S'S, (4) 


whence the term “square root of a matrix” is seen to correspond to the 
ordinary square root of an algebraic expression. [The identity of equation (4) 
with the fundamental theorem of factor analysis (12, p. 19; 15, p. 70) isa 
clear indication of why factor analysis independently discovered the square 
root method, although it was referred to ‘by various names.] In other words, 
the square root method applied to a matrix R yields a matrix S such that 
premultiplication by its transpose (i.e., column-by-column multiplication of 
S by itself) reproduces the matrix R. It is convenient, at times, to refer to the 
“square root operation,” by which is meant (s’)"*, since (S’)~* operating 
(premultiplying) on R produces S. Then the square root operation can be 
applied to other matrices than the basic one from which it is derived. 


3. Concepts and Notation in Multiple Group Methods 


The square root method was presented first so that the technique 
could be employed as needed in the development of the, multiple group 
methods of factor analysis. However, the notation of the preceding section will 
not be used in the remainder of this paper. Instead, a system of notation will 
be employed which is both clear and, when possible, closely related to that 
in the existing literature. The notation along with the basic concepts in 
multiple group factoring methods will be presented in this section. 

Of course the fundamental entity in factor analysis, aside from the 
scores themselves, is the matrix of observed correlation coefficients. In most 
methods of factor analysis, the reduced correlation matrix is used, meaning 
that the values in the principal diagonal are reduced from self correlations of 
unities to (estimates of) communalities. eee os 

The basic concept that distinguishes the methods of factor analysis 
under consideration is that of grouping of variables. Either by arbitrary or 
carefully selected grouping of variables a number of common factors can be 
extracted in one operation, and thereby a substantial reduction in the labor 
of computing residual matrices is realized. Some of the different points of 
view regarding the selection of groups of variables will be indicated in the 
final section of this paper. In any event, all the group factor methods have 
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the common attribute of expediting the total factor analysis by the selection 
of a number of linearly independent groups approximating the rank of the 
reduced correlation matrix. 

Except in rare circumstances, the common factors extracted in a single 
operation are oblique to one another. Therefore another basic concept.is that 
of the matrix of correlations among the factors. Also, since the factors are 
correlated, the immediate results of a group factor analysis must lead to two 
matrices—a factor pattern and a factor structure (12, p. 16). The first of 
these gives the coefficients of the factors in the linear descriptions of the 
variables, while the second gives the correlations of the variables with the 
factors. 

These results of the group factor analysis can be used to obtain a matrix 
of reproduced correlations; and hence the residual matrix can be determined. 
If the residual matrix is not sufficiently close to the null matrix then the group 
factor method can be applied, again, to the residual matrix. 

After it has been determined that the multiple group factor solution 
adequately reproduces the observed correlations, somé“Investigators may 
still consider such a solution as a preliminary step to the rotational problem 
(17, p. 171). In seeking “simple structure” by rotation of axes, the problem 
is simplified if an orthogonal frame of reference is first obtained. Hence, two 
additional concepts are introduced—an orthogonal factor matrix and the 
transformation matrix from the oblique to this orthogonal solution. The 
implications of stopping with the oblique solution obtained directly by the 
multiple group method or proceeding to the rotational problem will be 
brought out in the final section. 

The foregoing convepts, which arise in the multiple group methods of 
factoring, are summarized in Table 2. The symbol associated with each 
concept as it will be used in the following section is listed for ready reference. 
Also, the notation employed by the three principal contributors to multiple 
group methods of factor analysis is preser.ted to assist in making comparisons. 


4. Computations in Multiple Group Methods 


While there are apparent differences in the several presentations of the 
multiple group method of factor analysis—and several real differences in the 
generality and breadth of the theory—there is a basic technique underlying 
all of them. A systematic development of such a basic multiple group method 
is presented in this section. The procedure is illustrated with a 9-variable 
example, taken from Holzinger’s unpublished notes, “Detailed Outline of 
Simple Solution,” involving a sample of 696 cases, 12 tests, and 4 factors. 


a. Reduced correlation matriz.:The first problem is that of determining, 


good estimates of communalities. That problem is extraneous to the scope 
of this paper. [Various methods for estimating communality are discussed 
in (12, pp. 156-159 and 17, pp. 282-318),] No special justification for the 
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particular choice will be given. Actually, in the 9-variable example, the 
estimates were obtained by a single triad for each variable, i.e., the quotient 
of the product of the two highest r’s for a given variable by the correlation | 
between the two variables correlating highest with the given variable. The ss 
reduced correlation matrix for the nine variables is i 


.75 78 44 45 51 21 30 31] 
75 69 .72 52 53 58 23 32 .30 1 
78 72 75 AT? 48 54 28 37 37 | 
44 47 91 82 82 33 .33 31 | 
R=| 45 48 82 .74 .74 37 36 
51 58 54 82 .74 .74 35 38 38 | 
21 23 28 33 37 35 35 45 .52 
30 32 37 33 36 38 45 58 .67 . i 
31 30 37 31 36 38 52 67 


where the variables are assumed to be in sequence from 1 to 9, and the 
estimates of the communalities appear in the principal diagonal. 

b. Grouping of variables. The analysis begins with an appropriate group- 
ing of variables. Thurstone (17) stresses the arbitrariness of grouping the 
variables, while Holzinger (11) and Guttman (8) emphasize the desirability 
of very careful selection of variables in each group according to some a 
priori hypothesis. In the example it is assumed that the nine variables can 
be placed in three groups such that the common factors corresponding to them 

will adequately explain the data. The three groups, with their constituent 
variables, are as follows: 


G, (1, 2, 3), G, : (4, 5, 6), 8, 9). 


It may be of some interest to note that the tests in G, are of verbal content, 

G, arithmetic, and G; spatial relations. | 
c. Sums of correlations. The factor solution is obtained in several steps, ( 

with the computations in this step being preliminary to the actual factorial 

results. The sums of the correlations of each variable with the respective 

variables of each group are first required. These are given by the formula: 


ta = =1, (5) 


where the sum is on the index h and the symbol “h eG,” is read “‘h is a variable 
in group G, .” Formula (5) represents nm different sums. For the rsa, 7 
where n = 9 and m = 3, 
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Next, the sums of correlations among groups are obtained by means of 


the formula: 
tin» (j,k = 1, +++ , m) (6) 


where the summation is on the index 7 within each group, in turn. These 
nine sums appear in Table 4, where an immediate check is available from the 
symmetry property. 

d. Correlations among factors. As indicated above, the common factors 
obtained in a single operation of a multiple group method of analysis are 
oblique to one another. The factors are represented by vectors through the 
centroid (or, more generally, a weighted average) of the respective groups of 
variables. While the individual variables are in standard measure, the com- 
posites are not necessarily so. If the oblique factors are designated by y 
(k = 1, «++ , m), this means that the variance of y, is not unity but has the 
value 7',, as given in Table 4; and in general, Table 4 consists of the variances 
and covariances among the m = 3 factors. Then, from the theory of corre- 
lation between two composites (12, pp. 34-37), the correlations among the 
factors are given by: 


= (7 


and are recorded in Table 5. 
e. Oblique factor structure. The correlations of the tests with the factors— 


the oblique factor structure—can be obtained by application of the same 
theory. Any test z; is in standard measure while a factor +, is a composite of 
such variables and is not in standard form. The structure value 8,, is the 
correlation r,,,, and can be computed by the formula (12, p. 36): 


tin 
(8) 
Te 


The structure matrix S for the example is given in Table 6. 

f. Oblique factor pattern. To complete the solution in terms of correlated 
factors, the linear descriptions of the variables in terms of the factors are 
required as well as their correlations with the factors. The coefficients in these 
linear equations, i.e., the pattern values, are the coordinates with respect to 
the oblique (factor) axes of the points representing the variables. The factor 
pattern can be obtained from the known factor structure S and the — 
tions among the factors %, as follows (12, p. 327): 


P = Se". (9) 
The bulk of work implied in formula (9) is the determination of the inverse 


of ® (especially for a large number of factors). Either the Doolittle method 
or the square root method can be used to obtain the inverse and to syste- 
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matically carry out the matrix multiplication to produce the pattern matrix. 
This explicit computation will not be done for the illustrative example, but 
will be accomplished along with two other concepts as outlined in the following 
paragraph. 

‘g. Square root method in multiple group analysis. A very efficient scheme 
for the calculation of the oblique factor pattern, the transformation matrix, 
and the orthogonal factor matrix can be developed by application of the 
square root method. The basic square root operation will be performed on 
the matrix © of factor correlations, and the resulting “square root’ matrix 


‘will be denoted by A. Expressed symbolically, 


= A’A, (10) 


and the square root operator is (A’)~’. This operator when applied to the 


transpose of the structure matrix yields the desired orthogonal factor matrix, 
i.e., 

(A’)"'S! = F’, (11) 
Stated another way, the orthogonal factor matrix is obtained from the 
oblique factor structure by means of the transformation matrix A™', as 
follows: 


F = SA". (12). 


The mathematical proof of the foregoing expression as the transforma- 
tion from an oblique to a rectangular coordinate system will not be presented 
here. A brief description of this special type of transformation is in order, 
however. The given set of variables may be considered as points in the common 
factor space, for which sets of coordinates are F = [a,,] and P = [b;;] with 
respect to an orthogonal and an oblique frame of reference, respectively. 
Now, if the oblique coordinates are known, the special transformation 
desired is one in which the first axis of the new system coincides with the 
first oblique factor axis, the second is in the plane of the first two oblique 
axes and orthogonal to the first, etc. This transformation is accomplished 
directly by the square root method applied to the matrix of cosines of angular 
separations of oblique axes (namely, the correlation matrix ¢). The resulting 
transformation from oblique to the specified orthogonal coordinates is given 
by the matrix equation: 


F = Pa’. . (13) 
But, upon substituting the expression for P from equation (9) into equation 


(13), it follows that \ 
F = = = SA", 
which is precisely formula (12). Hence the transformation to the orthogonal 
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frame of reference (F) is described in terms of projections (s;,) on the oblique 
axes instead of the coordinates (b,,) in the original oblique reference system. 

As indicated above, the orthogonal factor matrix F as expressed in 
equation (11) can be obtained by application of the square root method. Also, 
by means of the square root method, the oblique pattern matrix P can be 
obtained without explicit computation of the inverse of @, which is required 
in formula (9). It follows from formula (13) that 


P’ = (14) 


and since each element in the right-hand member of this expression comes 
from the application of the square root method, the oblique pattern results 
from a simple matrix multiplication. 

The schematic formulation for the square root method to get the trans- 
formation matrix, the orthogonal factor matrix, and the oblique factor 
pattern is presented in Table 7A, and the application to the 9-variable 
example appears in Table 7B. In applying this procedure to a larger set of 
variables, the worksheet. might more conveniently be arranged to list the 
matrices S, F, and P in adjacent vertical blocks instead of their transposes 
in horizontal sections. 

h. Reproduced correlations. If the multiple group method of analysis is 
carried to the stage of an orthogonal factor solution, then the reproduced 
correlations can be obtained by the fundamental theorem of factor analysis, 


R' = FF’. (15) 


On the other hand, if the multiple group method of analysis is employed to 
get an oblique solution without rotation of axes, then equation (15) is not 
applicable. There are several formulas for getting the reproduced correlations 
directly from the component parts of an oblique solution. The basic formula’ 
(12, p. 19), involving the multiplication of three matrices, follows: 


R' = Per’, (16) 
Two other expressions for the reproduced correlations (12, p. 327) are: 
R' = PS’ = SP’, (17) 


which involve precisely the same kind of computation as formula (15). Still 
another formula is 


R' = So's’, (18) 

which is essentially the computation described by Guttman (8, p. 215). i 
Since both the orthogonal and oblique solutions were obtained for the | 

illustrative example, the reproduced correlations were computed by formula 
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.15) and checked by the second of formulas (17). The reproduced correlation 
matrix, including reproduced communalities, follows: 


[82 .74 .78 438 45 51 21 31 .30 
70 .72 53 52 57 24 31 30 


75 47 48 54 27 37 38 
93 82 81 34 32 31 
R' = 74 36 37 37], 
75 36 38 .38 
36.45 
58 .67 
78 | 
where the terms below the diagonal of the symmetric matrix were deleted j 


for simplicity. 


i. Residual matrix. The residual matrix, with k factors removed, is | | 
defined by ; | 
(19) 
For the 9-variable os the residual matrix is : 

00 O01 01° 00 

00 .00 | 

5. Comparison of Multiple Group Methods wa H 


One of the principal differences that appears in the several developments ; 
of multiple group methods of factor analysis really is concerned with the i 
formulation of scientific hypotheses rather than with the method of analysis. 4 
Guttman (8) and Holzinger (11) suggest that the multiple group methods be a 
used in conjunction with some a priori psychological theory. Guttman 
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emphasizes the fact that the computational procedures of multiple group 
methods can be applied in any event, but most psychological meaning can be 
gained only through the testing by the data of preconceived hypotheses, 
These hypotheses are reflected in the specific manner of grouping the variables 
and in the resulting common factors (usually oblique). 

‘On the other hand, Thurstone (17, pp. 171, 173) emphasizes that the 
multiple group method of factoring is quite independent of the manner of 
grouping the variables, and in an example, deliberately sets up groups in an 
arbitrary fashion, with litt!2 reference to the correlations. This thought is 
evident when Thurstone (18) calls attention to the unnecessary restrictions 
‘that Holzinger (11) placed upon the matrix of correlations in order to use 
his “simple method of factor analysis.” These restrictions are unnecessary 
when the object is simply to get a reduction of the correlation matrix to a 
factor matrix by the expedient multiple group method; they are not unneces- 
sary when the object is to test some specified hypothesis by use of multiple 
group analysis. 

Thus, while Holzinger considers the multiple group method suitable 
only if the correlation matrix is amenable to sectioning into portions of 
approximate unit rank, and Guttman prefers to group the variables so as to 
avoid or reduce the problem of rotation of axes, Thurstone conceives of the 
multiple group method primarily as another (efficient) technique for initial 
factoring to provide an orthogonal factor matrix, “which is the starting-point 
for the rotational problem” (17, p. 171). 

There is no doubt about the effectiveness of the multiple group methods 
in reducing the labor of computing residual matrices. Instead of extracting 
one factor at a time, and computing a residual matrix after each, the basic 
theorem (7, p. 12) underlying multiple group methods implies that only one 
residual matrix need be computed after extracting several factors at one time. 
If a number of linearly independent groups is selected equal to the dimension 
of the common factor space then only one residual matrix will be necessary; 
otherwise, if the first estimate of the number of clusters is too small, the process 
has to be repeated again. If too many clusters should be selected then the 
case of multi-collinearity will be evident in the matrix of correlations among 
the group factors (and the inverse will not exist). 

It has been pointed out that if the total number of common factors is 
not extracted in a single operation, then the multiple group method can be 
applied again to the residual matrix obtained after the first operation. And 
this can be repeated as many times as necessary to bring the residuals down 
to negligible values. In the successive application of the multiple group 
method, the common factors extracted at each stage are oblique to ‘one 
another, but the factors obtained in any single stage are orthogonal to’ all 
factors extracted in other stages. The implication of this is that if an @ priori 
hypothesis involves an oblique structure then all the common factors must 
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be extracted in one operation, or else subsequent rotation might be necessi- 
tated. 


Both Holzinger and Thurstone have considered only simple (unit- 


weighted) composites of variables in essentially non-overlapping groups. 


Guttman (8, p. 216) considers this case as the simplest, and usually quite . 


adequate, manner of grouping variables. However, by use of the weight 


matrix X, he presents the most general aaa to multiple group methods 


of factor — 


‘ 
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A SOLUTION FOR CASE III OF THE LAW OF COMPARATIVE 
JUDGMENT 


Raymonp H. Burros 
UNIVERSITY OF ILLINOIS 
AND 
W. A. GrBson 


UNIVERSITY OF NORTH CAROLINA 


A solution for Case III of the Law of Comparative Judgment, modeled 
after Thurstone’s solution for Case IV but eliminating the restrictive assump- 
tion of relatively equal discriminal dispersions, is developed. 


Case III of the Law of Comparative Judgment (6, p. 280) may be written 


as follows: 
S. — S; = +03, (1) 
where 
S, = scale value or average momentary estimate of the position of 
stimulus k on the psychological continuum, 
S; = scale value or average momentary estimate of the position of 
stimulus 7 on the psychological continuum, 
o, = discriminal dispersion or standard deviation of the distribution 


of momentary estimates of the position of stimulus k on the 
psychological continuum, 

o; = discriminal dispersion or standard deviation of the distribution 
of momentary estimates of the position of stimulus 7 on the 
psychological continuum, and 

X;,, = normal deviate corresponding to the proportion of times that 
stimulus k is chosen instead of stimulus 7 when the two are 
compared. 


The assumptions leading to Case III (6, pp. 273-280) may be stated as 
follows: 


(a) Whenever a subject is about to choose one of two stimulus objects 
on some basis controlled by the experimenter, each object arouses 
its own momentary estimate of the true scale value of that stimulus 
on the uni-dimensional psychological continuum which is the basis 
of the choice. 


57 


ni 


58 PSYCHOMETRIKA 


(b) Of two stimuli being compared, the one which has the largest momen- 
tary estimate is the one chosen by the subject. 

(c) The distribution of differences between the momentary estimates 
for any two stimuli is normal in form. (A careful following of Thur- 
stone’s development shows that this is the only normality assumption 
that is required, although it has been customary to assume that the 
distribution of momentary estimates for each stimulus is normal 
in form.) 

(d) The correlation between the momentary estimates for any two 
stimuli is zero. 


Equation (1) could be written explicitly for all values of j and k to form 
a set of simultaneous equations in the S and o values. Probably because of 
their non-linear form, however, no solution for this set of simultaneous equa- 
tions has appeared. A section in Guilford entitled “Solution by Case IIT” 
(4, pp. 233-234) may seem to imply that such a solution already exists, but the 
text of this section indicates otherwise. The heading merely refers to the 
determination of the scale values from equation (1) after the discriminal 
dispersions have been estimated by other means. 

By making use of the additional assumption that 

(e) The discriminal dispersions are all of the same order of magnitude, 
Thurstone (6, pp. 280-281) has developed what is called Case IV of the Law 
of Comparative Judgment, as follows: 


Since equation (2) is a homogeneous linear equation in the S and o values, 
any method of solving such equations could be applied to reach a solution for 
Case IV. A least squares solution has recently been provided by Gibson (3). 
Thurstone (7, pp. 293-296) earlier worked out an ingenious and rapid solution 
based on a consideration of the analytic geometry involved in the plots of 
columns of the square matrix of X values. His estimate of o; is 


on = (a/V,) — 1, (3) 
where 
2N 
a= (4) 


V, being the standard deviation of the entries in column k of the matrix X 
(containing the X values). His unit of measurement is defined as follows: 


La =N. (5) 


. 

¥ 
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The solution for Case III to be developed here is modeled after 
Thurstone’s solution for Case IV but does not involve the restrictive assump- 
tion of relatively equal discriminal dispersions. A recent article by Burros (2) 
presents a solution for a combination of Cases III and IV which nevertheless 
retains the assumption of approximately equal discriminal dispersions. 
This is shown by Burros’ use of his equation (17) in the development of his 
solution. It, too, is modeled after Thurstone’s solution for Case IV. Burros’ 
estimate of o, is ' 


= (6) 


where 


N 


and V, is defined as before. His unit of measurement is given by equation (5). 

For the development of a solution for Case III, consider any three 
stimuli, h, j, and k. Let h and j be held constant, once chosen, and let k vary. 
Then equation (1) holds for stimuli j and k. A similar equation may be written 
for stimuli h and k: 


Si = XuVoi toi. (1a) 


Substracting equation (la) from equation (1) and rearranging terms, we 
obtain, 


8 
Ay = k i. 8 


Equation (8) is linear in X,; and X,, for a fixed value of k. Hence the 
plot of column j against column h from the matrix X will be approximately 
linear. Deviations from strict linearity are attributable to the imperfect 
applicability of Case III to the empirical data, sampling error, and variations 
in o, as k takes on different values. 

Now some kind of average of the expression 


m = Maton (9) 
Vo.t+ 


over all values of k, should represent the slope of the best-fitting line for 
this set of points. Since the choice of a particular measure of central tendency 
is essentially arbitrary, a convenient one for the present purpose, though a 
little unusual, is the square root of the ratio of the arithmetic mean of the 
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squared numerators to the arithmetic mean of the squared denominators. 
Since h and j are held constant while k varies, the result is then 


1 
m= 
1 2 2 
Date 


The choice of this peculiar measure of central tendency is really more 
than a matter of convenience; it is essential to the present solution for Case 
III. This alone justifies its use here. Other considerations lose much of their 
importance when it is recognized that this average usually will not deviate 
markedly from the more conventional ones. An added guarantee of this is 
the fact that o, occurs in both numerator and denominator of equation (9), 
thus inhibiting wide fluctuations in the size of the fraction from one value of * 
k to the next. 

Equation (10) can be further simplified by defining the unit of measure- 
ment for the scale in the following way: 


=N. (11) 


(10) 


Then the slope of the best-fitting line is 


Vic+oa; 
(12) 
Vita; 

The plot of column j against column h of the matrix X could actually 
be constructed and the slope of the best-fitting line determined by any fitting 
method. Thurstone (7, pp. 293-294) construed this plot as representing a 
scatter diagram, with the slope of the line of best fit being simply the geometric 
mean of the slopes of the two regression lines. This leads to the following 
formula for the slope of the best-fitting line: 


m= V,/V,, (13) 


where V, represents the standard deviation of the entries in column j of the 
matrix X and V, represents the standard deviation of the entries in column h 
of the same matrix. Since equation (13) is essentially the ratio of two indices 
of dispersion, the V’s might be less restrictively defined as being any reason- 
ably stable measure of variation, such as the average deviation. 
Equations (12) and (13) give 
ff 2 
Vis (14) 


Vite 


Hence, 


VVite=ViV1 +07, (15) 
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so that 


Vil + = Vi(1 + (16) 


Since equation (16) holds for any values of h and j, either side may be 
set equal to a constant defined as b. Then, for stimulus k, 


+ = (17) 


Equation (17) was previously employed by Burros (2, p. 64). 
Now if equation (17) is written in the form 


(b/Vi) =1 +o (18) 
and then summed over k, the result is 
=N+ Lie. (19) 


By virtue of equation (11), equation (19) can be further simplified and then 
solved explicitly for b to give 


(1/Vi)’ 


Note that b is twice the harmonic mean of V;,. Equation (18) can be written 
in the form 


(20) 


om = (b/Vi) — 1 (21) 


to give the Case III estimate of «7 once b has been determined. 

The method of solution for the discriminal dispersions is then quite 
direct. Compute the squared standard deviation (or the square of some other 
stable measure of dispersion) for the entries in each column of the matrix X, 
and then determine its reciprocal. Sum these reciprocals and substitute the 
result into equation (20) to determine b. Substitute b and the appropriate 
Vi into equation (21) to get oj, . 

These oc; estimates must now be used to determine the scale values. An 
efficient method for doing this will be given here. First form the matrix Q, 
with the general entry 


= + 9; (22) 


by bordering the space to be occupied by Q with an extra column and row 
containing the oj estimates. Then each entry in Q is simply the sum of the 
o; estimates that head the row and column which intersect in that entry. 
The kth diagonal element in Q is 20; . A convenient summational check on 
the correct formation of column j in Q is to be had from the equation 


= ot + No} = N + No} = N(1 03), (28) 
& k 
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which simplifies by virtue of equation (11). The next step is to form a matrix 
R with the general entry 


= = Vo + o; (24) 


There is no summational check here, but of course each entry in R, when 
squared, should yield the corresponding entry in Q. The next step is to form 
the matrix S, whose general entry is given by equation (1) and is obtained 
from the matrices X and R by multiplying the kj entry in X by the corre- 
sponding entry in R. There is no summary check here. It should be pointed 
out explicitly that the symmetry (or skew-symmetry) of the matrices X, 
Q, R, and S permits a substantial reduction (by almost half) of the work 
that would otherwise be required to form these matrices. 
Now the sum of column j in the matrix S is 


8) = — NS; . (25) 


Since the origin of the scale has not yet been defined, it is convenient to fix 
it as follows: 


S, = 0. (26) 
Equation (25) then reduces to 
> — = —NS;. (27) 
Hence, 
8, = (& 8). (28) 


Thus the scale values themselves are obtained simply by multiplying through 
the column sums of the matrix S by the quantity (—1/N). A check here is 
that, by equation (26), the sum of the resulting scale values should be zero. 

The origin and unit of measurement for the resulting scale are defined 
by equations (26) and (11), respectively. Any desired change of unit or origin 
can be effected by appropriate modifications of the scale values and dis- 
criminal dispersions. The unit of measurement can be altered by applying a 
constant multiplier to both the scale values and the discriminal dispersions. 
A change of origin (to make all scale values positive, for example) can be 
brought about by adding a suitable constant to all scale values without 
changing the discriminal dispersions. 

This solution for Case III requires that the matrix X be completely 
filled, with no blanks occurring in the side entries, and with zeros (presumably) 
in the diagonal cells. If some entries in an X table are missing because of 
the instability of normal deviates for very high or very low proportions, 
the table can usually be rearranged and then subdivided into a number of 
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overlapping square blocks in such a way that each block is complete and 
skew-symmetric. Then this solution can be applied to each block separately 
and the results for all blocks combined by some fitting procedure. These 
remarks apply also to the Case IV solution of Thurstone and the Cases 
III-IV solution of Burros. 

A pre-publication reviewer has called attention to a possible empirical 
difficulty by exhibiting a fictitious example in which the present Case III 
solution yields some negative o;, values. Equations (21) and (20) indicate 
that a negative o}, necessarily arises whenever any V; exceeds the value of 
b, or whenever the reciprocal of any V; is less than half the arithmetic mean 
of those reciprocals. Equations (3) and (4) indicate a completely analogous 
state of affairs with Thurstone’s solution for Case IV. His solution will neces- 
sarily vield a negative o, estimate whenever any 1/V, is less than half the 
mean of such reciprocals. Since the rank ordering of the various values of 
1/V;, is the same as that for 1/V, , absurd results are likely to occur for the 
same stimuli by both of these solutions. They may be more serious with the 
Case III solution because the stronger negative skewing of the distribution 
of 1/V; may make it easier for the tail of that distribution to extend below 
half its mean. 

A fictitious example which yields absurd results by both of these solutions 
is shown in Table 1. Table 2 shows the corresponding X matrix, as given by 
equation (1) solved explicitly for X,; . Table 3 gives the results of applying 


TABLE 1 


A Fictitious Paired Comparisons Example 


Stimulus 


1 2 3 4 5 
- .24 - .24 - .04 
k 2.00 1,50 1,00 50 
TABLE 2 TABLE 3 
The Fictitious X Matrix Three Thurstone-Type Solutions 
Stimulus Stimulus 
1 2 3 4 5 1 he 3 4 5 
000 000 2133 400 1, 200 v, 2034 .0125 .0192 0368 1762 
2 000 000 080 179 291 - .146 2,444 1,778 1,007 ,083 
é 3 - .133 = .080 000 253 -.711 3.711 2,065 666 
34 400 000 179 ick 2.356 1,532 800 167 
5 “1,200 -.292 =.253 .179 .000 
a = .3850 b = ,05888 c = ,02944 


to Table 2 the Case IV solution of Thurstone (¢,,), the present Case III 
solution (;,) and the Cases III-IV solution by Burros (¢,,). The first two 
solutions give absurd results for the same two stimuli, the relative magnitude 
of the negative values being larger in the Case III solution. Both of these 
solutions apparently suffer from the same basic defect, namely, that small 


64 PSYCHOMETRIKA 


positive true values may be estimated as negative by the fitting procedure 
that is used. 

The last row in Table 3 contains a simple answer to this dilemma, if it 
should ever arise with empirical data. None of the Burros estimates of o; 1s 
negative, and equations (6) and (7) show that they never can be negative. It 
is therefore recommended that, in the event of negative o; estimates by the 
present Case III solution, the Burros solution be used. Equations (7) and 
(20) give 

b 


c=- 


(29) 


and equations (6), (29), and (20) yield 
ocx = 3(0/Vi) = 3[(b/Vi — 1) + 1] = 3%, + 1). (30) 


Hence all that is involved in switching from the present solution to that of 
Burros is to add 1 to each o}, and divide by 2 to get the corresponding o,, , 
all of which will be positive real numbers. The only changes in the subsequent 
computations for the scale values, as they have been outlined here, is that 
the last two steps in equation (23), a check formula, are invalid because 
equation (11) is then replaced by equation (5) as the definition of the unit of 
measurement. The first equality in equation (23) still holds, however, so 
that a summational check still exists at that stage. 

This Case III solution has been applied to several different sets of 
empirical data without yielding absurd results. These researches include a 
traditional psychophysical experiment (5), a study of nationality preference 
(4, p. 227), and two studies of reaction potential (1 and 2). It is the writers’ 
belief that absurd results will occur only rarely with empirical data, so that 
this solution should have fairly wide applicability in psychological research. 
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THE PREDICTED AND OBSERVED EFFECT OF CHANCE 
SUCCESS ON MULTIPLE-CHOICE TEST VALIDITY* 


LYNNETTE B. PLUMLEE 
EDUCATIONAL TESTING SERVICE 


Assuming chance to be fully operative, the predicted effect of chance 
success on test validity when answer options are supplied depends on the 
number of options, the difficulty of the test and the variance of test scores. 
Predicted validity values are compared with empirical validity values in an 
experiment which used the same mathematics test items with and without 
answer options. 


The present paper is an extension of an earlier study on the effect of 
chance success on item and test statistics when answer options are supplied. 
The earlier study (3) compared values predicted by Guilford’s formula for 
item difficulty (2), Carroll’s formula for test reliability (1), and the author’s 
formula for item-test correlation with values obtained in an empirical study 
using mathematics achievement test material. The present paper presents the 
equation for predicting the effect of chance success on test validity and com- 
pares predicted values with empirical values from the same source of data 
as that used for the previous study. 

In deriving the prediction equation, the assumptions made in the earlier 
study will apply here. They are: 

1. Every examinee attempts every item in the test. 

2. Every examinee who knows the correct answer to an item answers the 
item correctly in both multiple-choice and answer-only form. 

3. Every examinee who does not know the correct answer to an item 
answers the item incorrectly in answer-only form and chooses from among the 
options on a basis of chance alone in multiple-choice form. 

4. The number of options per item is the same for all items in the multi- 
ple-choice form of the test. 


*Part of a paper presented at the 1952 meetings of the APA. The author acknowledges 
with appreciation the helpful advice of Professor Harold Gulliksen and the joint support 
4 the project by the College Entrance Examination Board and the Educational Testing 

rvice. 
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The following notations will be used: 
Multiple- Answer- 


Choice Only 
Form Form 
of Test of Test 
R Probability of answering an item correctly on the basis of chance 
alone, where R = 1/number of answer options. 
W Probability of answering an item incorrectly on the basis of 
chance alone, where W = 1 — R. 
h h Number of items in the test. 
a x Score (number of items correct) on the test. 
a’. Zk Score of the kth individual on the test. 
Yk Yk Score of the kth individual on the criterion. 
t t Number of individuals who take the test. 
MM: M, Mean score on the test. 
M, M, Mean score on the criterion. 
Ox’ Oz Standard deviation of scores on the test. 
Oy Cy Standard deviation of scores on the criterion 
Te'y ey Correlation between scores on the test and scores on the criterion. 


Carroll (1) has given formulas for the expected multiple-choice mean and 
standard deviation, which can be expressed in terms of number-right scores 
as follows: 


M,. = WM, + Rh, (1) 
and 


= VW'o2 + RWh — M,). (2) 
It will be seen from (1) that 
= W > + Rht. (3) 
k=1 k=1 


If multiple-choice scores on the test for those individuals making a given 
score on the criterion behave according to large sampling expectations, it 
may be demonstrated that 


> = W > + Rh (4) 
k=1 k=1 =1 


Substituting (1), (2), and (4) into the formula for the correlation between 2’ 
and y, 


1 t 


Toy = (5) 


t t t | 
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it will be seen that 
t 
k=1 


VW? + RW(h — 


It will be noted that the numerator of (6) is equal to Wr,,c,0, , and hence 


(6) 


Te'y => 


Wr.,0- (7) 
+ RW(h — M,)’ 
or 
Tey R{(h-M, 
Vary o. ) 8) 


It will be seen from equation (8) that the validity of the multiple-choice 
form of a test is theoretically always less than the validity of the same test 
in answer-only form. As oz becomes very large relative toh — M, or as W 
approaches 1, 


lim “2% = 1. 
Thus, if the mean score for the answer-only form of a standardized test is equal 
to about one-half the number of items and the standard deviation of scores is 
equal to about one-fifth the number of items, then the multiple-choice validity 
will theoretically approach the answer-only validity as a limit as the number 
of items and/or the number of answer options becomes large. 

As described in the earlier study, each of four 36-item sections of mathe- 
matics achievement material was prepared both in multiple-choice form (with 
five answer options) and in answer-only form. An additional 16-item section 
was prepared which contains both answer-only and multiple-choice items. 
These five sections were combined into four tests in the manner indicated in 
Table 1. The four 36-item sections in multiple-choice form are designated as 


TABLE 1 
Arrangement of Tests 
Time Number Test W Test X Test Y Test Z 
(min- of 
utes) Items (N = 125) (N = 1380) (N = 128) (N = 126) 
Part I 15 16 Set of items, common for all tests, with odd-numbered items in 
answer-only form and even-numbered items in multiple-choice 
form. 
PartII 35 36 Section M1 _— Section A3 Section M3 _—— Section Al 
Part WI 35 36 Section M2 Section A4 Section M4 Section A2 


PartIV 35 36 Section A3 Section M1 Section Al Section M3 
Part V 35 36 Section A4 Section M2 Section A2 Section M4 
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M1, M2, M3, and M4; the same sections in answer-only form, as Al, A2, A3, 
and A4, respectively. Thus, Tests W and X, and again Tests Y and Z, differed 
only in the order in which the multiple-choice and answer-only sections were 
administered. 

In order to compare the extent to which the predicted effects of chance 
success on test validity agree with results obtained in practice, rank standing 
in first-year mathematics and total academic standing in all courses for the 
first year were obtained for the subjects used in the study. Validities were 
computed against both criteria for the 509 subjects for whom complete data 
were available. Results are shown in Table 2. 

It will be noted that equation 8 tended to over-correct and that values 
obtained by applying equation 8 to the observed answer-only values are no 
nearer than the observed answer-only values themselves to the observed 
multiple-choice values. Moreover, the observed answer-only validity values 
are quite similar to observed multiple-choice validity values; no statistical 
difference was found among the 16 validity coefficients for either criterion. 
Although the differences between predicted and obtained multiple-choice 
validity values are not great, the direction of the differences appears to be 
significant by a simple signs test. From the data in Table 2, then, it cannot 
be concluded that the multiple-choice form of test predicts less well than the 
answer-only form. 

One may conclude, therefore, on the basis of the current investigation, 
that: 

1. Theoretically, supplying options for a test decreases its validity. 

2. For the mathematics achievement tests used in the current study, the 
multiple-choice form was as effective as the answer-only form for predicting 
success in coursework, even though the tests were fairly short. 

It will be noted that the findings in the current study regarding validity 
thus correspond to those regarding reliability in the earlier study. As in the 
case of multiple-choice test reliability, some explanation for the deviation of 
observed multiple-choice validity from predicted multiple-choice validity 
may be found by considering the extent to which the assumptions basic to 
the prediction formulas were met in practice. 

The fourth assumption was met; all items had five options. Although the 
first assumption was not met, a comparison of item statistics for items not 
reached by examinees with those reached by most examinees revealed no 
differentiating pattern which might influence the validity results. 

The third assumption was likely not met since examinees made careless 
errors; the greater likelihood that such errors will be detected in multiple- 
choice form and rectified may contribute to increased validity for the multiple- 
choice answer form. 

However, the assumption most likely to be seriously violated in a well- 
prepared test is that chance alone influences the response made by the 
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examinee to the multiple-choice item. For many examinees, some multiple- 
choice items are equivalent to the same item in answer-only form, since their 
wrong answer is among the presented options. Furthermore, partial credit 
may be given for partial knowledge since the examinee with some knowledge 
may be able to limit his selection to a choice among two or three options. Both 
of these factors would operate towards yielding scores in the multiple-choice 
form more nearly like those obtained in answer-only form and would hence 
contribute towards increased validity. 
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A SECOND-ORDER FACTOR ANALYSIS OF REASONING 
ABILITIES* 


LEONARD MatTIN AND Dorotuy C. ADKINS 


THE UNIVERSITY OF NORTH CAROLINA 


This article reports a second-order factor analysis of the 13 interpretable 
first-order factors of Adkins and Lyerly based on 66 variables for 200 Army 
men. Ligne tore to simple structure for 6 second-order factors was un- 
usually good. Loadings of the original 66 variables on the second-order factors 
were also obtained. Five of the factors are tentatively interpreted as Precision 
in Formation and Use of Verbal Concepts, General Verbal Fluency, Visualiz- 
ing Spatial Constancy During Movement, Speed in Analysis, and Flexibility 
in Analysis. 


In a factor analysis of the scores on 66 variables of 200 Army men, 
. Adkins and Lyerly (1), attempting to clarify the nature of reasoning factors, 
obtained 13 interpretable oblique factors. This article reports the finding 
of 6 second-order factors from an analysis of the correlations between the 
13 first-order factors. 


The First-Order Factors 


The 13 first-order factors, together with the tests that had loadings of 
.30 or over, presented in order of decreasing size, were as follows: 

A’, Perception of Abstract Similarities (Verbal Classification II, Verbal 
Classification I, Figure Analogies, and Word Squares). This factor was 
described as the ability to perceive a type of similarity important in abstract 
thinking or as an ability to educe relations of similarity (1, 72-75). 

B’, Hypothesis Verification (Progressive Matrices C, Progressive Matrices 
E, Verbal Analogies, False Premises). This factor was interpreted as the 
particular aspect of inductive thinking required in testing or verifying 
hypotheses (1, 76-80). 

C’, Verbal Relations (Reading, Sentence Order, Vocabulary, Absurdities, 
Reading II, Practical Situations). The high loadings of Sentence Order and 
Absurdities make evident that this factor involves more than mere vocabulary, 
though the ability is undoubtedly identical to the factor commonly called 
“Verbal” (1, 64-65). 

D’, Perceptual Speed (Forms, Block Counting, Progressive Matrices E, 
Identical Forms, Overlapping Circles). This factor agrees reasonably well 

*This manuscript represents a condensation of a thesis presented by Mr. Matin 


for the degree of Master of Arts at The University of North Carolina, Miss Adkins having 
been chairman of the advisory committee. 
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with French’s (3) description of Perceptual Speed, in which is stressed the 
task of finding quickly a given configuration in distracting material (1, 
65-67). 

F’, Flexibility of Perceptual Closure (Figure Classification IIA, Figure 
Classification IIB, Map Planning, Camouflaged Outlines, Identical Forms). 
Although this factor did not correspond fully with any previously identified 
factor, the best description was in terms of closure against distracting material 
(1, 80-85). 

H', Deduction (False Premises, Identical Forms, Word Squares). This 
factor was interpreted as the ability to draw correct inferences (1, 85-86). 

J’, Number (Numerical Operations II, Numerical Operations I, Number 
Series, Arithmetic, Painted Blocks, Circles, Letter Series). This factor was 
identified as the familiar Number factor (1, 68). 

K’', Word Fluency (Suffixes, First and Last Letters). The Word Fluency 
factor was described as the ability to produce quickly words that meet 
certain more or less mechanical specifications (1, 68-69). 

L’, Space 1 (Figures, Cards, Block Counting, Geometrical Puzzles). 
This factor corresponds closely with the factor that Thurstone calls Space 1, 
which seems to involve visualization of a rigid configuration when it is moved 
into different positions (1, 69-70). 

M', Concept Formation (Picture-Group Naming, Word-Group Naming, 
Verbal Analogies). The tentative hypothesis was that this factor represents 
the ability to formulate abstract or precise verbal concepts (1, 86-89). 

N’, Speed of Perceptual Closure (Street Gestalt Completion, Identical 
Forms, Mutilated Words, Mutilated Pictures). This factor corresponds to 
Thurstone’s Speed of Closure factor, interpreted as the ability to organize 
a chaotic visual field into a single percept (1, 70). 

O', Ideational Fluency (Topics, Things Round, Reasons). This factor 
appears to be the factor termed Ideational Fluency by other investigators 
and interpreted as the ability to produce ideas rapidly, regardless of their 
quality (1, 70-71). 

P’, Space 2 (Mechanical Information, Mechanical Movements, Practical 
Situations). This factor Adkins and Lyerly termed Space 2 (interpreted by 
Thurstone as the ability to visualize a configuration in which there is move- 
ment or displacement among its parts) only with some doubt, suggesting 
that the hypothesis that the variance in the factor was accounted for by 
“mechanical experience” had not been ruled out (1, 71-72). 

The inter-correlations among the 13 first-order factors, which are re- 
produced below the main diagonal of Table 1, were obtained from the cosines 
of the first-order reference vectors (1, 122). By the complete centroid method 
(5, 161-170), six second-order factors were extracted, the factoring being 
repeated four times until communality values stabilized. The sixth-factor 
residuals are reproduced in Table 1 above the main diagonal. Projections of 
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the first-order factors on the centroid reference vectors appear in Table 2. 
Rotation to oblique simple structure was accomplished by the single-plane 
method, with final adjustments by the radial method (5, 216-224, 194-216). 
The transformation matrix and the final oblique factor matrix are presented 
in Tables 3 and 4, respectively. It may be noted that the approach to simple 
structure, according to Thurstone’s criteria (5, 335), appears to be unusually 
close for a second-order factorization. The reference vector cosines appear 
in Table 5. Although these cosines indicate appreciable correlation among 
some of the second-order factors, the authors are not inclined to suggest 
that a third-order analysis be performed, at least not until more is known 
about second-order analyses. 

Some reviewers have expressed skepticism about determining as many 
as six factors from 13 variables. It may be noted, however, that as many as 
eight factors can be determined by 13 variables (5, p. 294). It should also be 
reported that the authors have explored the possibility that as few as four or 
five factors would be sufficient in this case, but that the residuals remained 
too large to be ignored. It is true, however, that one of the six factors, Factor 
V, has only one high positive loading and one fairly high negative loading. 
Although no interpretation is suggested for this factor, it can scarcely be 
dismissed as a residual factor. 


Interpretations of Second-Order Factors 


In the brief interpretations of the second-order factors, loadings ex- 
ceeding .25 will be indicated. Some cognizance was also taken of the loadings 
of the original 66 variables on the second-order factors. 

Although not a great deal of discussion will be devoted to the bearing 
of these original test loadings on the interpretation of the second-order 
factors, the complete table is presented in Table 6. No attempt was made to 
rotate the second-order factors to a satisfying fit in the test space. In general, 
interpretations of the second-order factors from inspection of the original 
test loadings are in essential agreement with those made from analysis of 
loadings of first-order factors. The loadings on second-order factors tend 
to hover around zero for tests that are significantly loaded on first-order 
factors that themselves have negative loadings on the second-order factors. 
In several instances, also, a test will have a possibly significant loading on 
one first-order factor that has a positive loading on a second-order factor, 
and also a possibly significant loading on another first-order factor that 
has a negative loading on the same second-order factor. The suggested in- 
terpretation of the latter finding is that the second-order bipolar factor 
represents two abilities, one at each pole, and that a subject may use either 
of the two alternative abilities in responding to such a test. Each bipolar 
factor has been named after the positive pole. 

Factor I, Precision in Formation and Use of Verbal Concepts, has a loading 
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of .61 for C’, Verbal Relations; .51 for M’, Concept Formation; and —.45 
for D’, Perceptual Speed. This factor seems to involve the ability to form 
and use verbal concepts precisely. The tests of both Verbal Relations and 
Concept Formation involve the analytic manipulation or the eduction of 
verbal concepts. Thus the fact that several tests of Verbal Relations did 
not depend upon high-level vocabulary suggested an interpretation stressing 
the perception and manipulation of verbal relations rather than merely 
understanding the meaning of words. 

Here, as in the other bipolar second-order factors to be reported, those 
tests with significant loadings on the first-order factor which is at the negative 
pole have near-zero loadings on the second-order factor. Both Forms and 
Progressive Matrices E, which have their highest loadings on Perceptual 
Speed, also have loadings slightly above .25 on Concept Formation, which is 
negatively correlated with Perceptual Speed (—.11). The interpretation may 
be that the two poles involve alternative or even antagonistic processes, 
either of which is used by some subjects in the performance of these tests. 

Since all the tests with loadings on this second-order factor greater than 
.25 (except Street Gestalt Completion, with a loading of .28) also have 
significant and positive loadings on either Verbal Relations or Concept 
Formation, the interpretation of the second-order factor based on loadings of 
the first-order factors at the positive pole is substantiated by consideration 
of loadings of the original tests on the second-order factor. 

Factor II, General Verbal Fluency, has a loading of .72 for Factor K’, 
Word Fluency; .52 for O’, Ideational Fluency; and .47 for A’, Perception of 
Abstract Similarities. 

The high loadings on this factor of Word Fluency and Ideational Fluency 
suggest a more generalized fluency ability. In the tests that were loaded on 
Perception of Abstract Similarities, too, a type of fluency in educing possible 
relations to be tested for similarity seems to be essential. The restriction of 
the task set for the subject is apparently least for the tests of the first-order 
factor with the highest loading on the second-order factor, Word Fluency, 
and greatest for the one with lowest significant loading, Perception of Abstract 
Similarities, thus permitting greater involvement of a fluency factor in the 
tests of Word Fluency than in the tests of Perception of Abstract Similarities. 

A large number of the original tests have loadings above .25 on this 
general fluency factor. Since nearly all with loadings above .40 very clearly 
entail the use of words, perhaps the factor should be termed General Verbal 
Fluency. 

Factor IIT, Visualizing Spatial Constancy During Movement, has a load- 
ing of .55 for Factor L’, Space 1; .46 for P’, Space 2; and — .32 for O’, Idea- 
tional Fluency. If the interpretation of Space 2 is tenable, the best hypothesis 
for this factor is that it calls for visualization involving maintenance of a 
framework containing parts that can move conceptually while, at the same 
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time, certain relations among the parts remain constant. In the reference 
tests for Space 1, rigid configurations are visualized as moving against a 
homogeneous background. In Mechanical Movements, which is highly 
loaded on Space 2, the “framework”’ is part of the object against which other 
parts move in relation to each other. Here the subject must maintain a con- 
ceptual object constancy while visualizing movement. 

Although there is no readily apparent reason for the negative loading 
of Ideational Fluency on this factor, it may have arisen from the loadings 
for Topics of .25 on Space 1 and of .52 on Ideational Fluency, which corre- 
lated —.41 with Space 1. Again, then, it is suggested that a bipolar second-order 
factor may mean simply that certain tests can be solved by alternative or 
antagonistic abilities. 

Factor IV, Speed in Analysis, has a loading of .61 on J’, Number; .61 
on A’, Perception of Abstract Similarities; .52 on D’, Perceptual Speed; and 
.33 on L’, Space 1. 

The high loadings on Number and Perceptual Speed immediately suggest 
some type of speed factor, apparently quite general in character as indicated 
by the diversity of the four first-order factors. Speed, however, may be 
simply an indicator of a high level of a given ability rather than a separate 
ability in itself. On this basis, some more fundamental ability common to 
these factors is suggested. The most plausible suggestion concerns analytical 
ability, possibly at the perceptual level. Perception of Abstract Similarities 
and Perceptual Speed are obviously analytical in nature. Space 1 requires 
the separation of parts of a perceptual gestalt. Some of the tests of Number, 
the two series tests, and Circles also call for analytical ability. While analysis 
does not seem to be necessary for simple arithmetic computations, analytic 
ability may be important in the original learning of arithmetical operations. 

Twenty-eight of the 30 tests with loadings above .25 on one of the four 
first-order factors that have high loadings on the second-order Factor IV 
also have loadings above .25 on it. Twenty-six additional tests, or 54 in all, 
have loadings above .25 on Factor IV. Such a large number of highly loaded 
tests gives the factor the appearance of a general factor; yet such an interpre- 
tation is scarcely consistent with the loadings of the first-order factors alone. 
Pending further clarification, it seems preferable to restrict the interpretation 
to that based on the loadings of the first-order factors, which resulted from 
rotation to simple structure. 

Factor V has a loading of .60 for H’, the first-order Deduction factor, 
and —.37 for N’, Speed of Perceptual Closure. Since only one positive load- 
ing, that for Deduction, exceeds .25, no attempt will be made to name this 
factor. The deductive factor was not over-determined in the first-order 
analysis, a fact which weakens the significance of findings relating to the 
second-order factor. 

The Identical Forms test, with a loading of .31 on Deduction and .48 
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on Speed of Perceptual Closure, may account for the negative loading of the 
latter first-order factor on Factor V. Again, the two first-order factors con- 
cerned are negatively correlated, —.29. As before, probably some subjects 
use one of the abilities and some the other. 

Factor VI, Flexibility in Analysis, has a loading of .45 for F’, Flexibility 
of Perceptual Closure; .36 for A’, Perception of Abstract Similarities; and 
—.25 for N’, Speed of Perceptual Closure. 

The positively-loaded first-order factors suggest that Factor VI involves 
analytic ability in some manner. The inductive nature of Perception of 
Abstract Similarities was discussed in the report on the first-order analysis 
(1, 73). Other recent studies have shown a relation between the Inductive 
and Flexibility of Closure factors (2, 4). 

The tests of Perception of Abstract Similarities and those of Flexibility 
of Perceptual Closure resemble each other. Nearly all the classification and 
analogies tests in the original battery are loaded on one or the other of them. 
The analytic nature of the abilities required by these tests is apparent, both 
factors involving the finding of common elements in distracting materials. 

In Botzum’s study, Flexibility of Closure and Speed of Closure were 
both positively loaded on the same second-order factor (2). Pemberton found 
the two almost orthogonal and not appearing on the same second-order factor 
(4). These findings are neither consistent with each other nor with the findings 
of the present study, in which the two closure factors correlate —.19 and 
appear, one with a positive loading and one with a negative loading of border- 
line significance, on the same second-order factor. The apparent bipolarity 
that was found in this study probably resulted from the fact that Identical 
Forms and Camouflaged Outlines had loadings slightly greater than .25 on 
both factors, which themselves were negatively related. It should be noted 
that since the identification of the Flexibility of Closure factor in the first- 
order analysis was not very certain, the interpretation of this second-order 
factor is still highly tentative. 


REFERENCES 


1. Adkins, Dorothy C. and Lyerly, Samuel B. Factor analysis of reasoning tests. Chapel 
Hill, N. C.: Univ. North Carolina Press, 1952. 

2. Botzum, W. H. A factorial study of the reasoning and closure factors. Psychometrika, 
1951, 16, 361-386. 

3. French, John W. The factorial composition of aptitude and achievement tests. Psycho- 
metric Monogr., No. 5, 1951. 

4. Pemberton, Carol. The closure factors related to other cognitive processes. Psycho- 
metrika, 1952, 17, 267-288. 

5. Thurstone, L. L. Multiple-factor analysis. Chicago: Univ. Chicago Press, 1947. 


Manuscript received 5/28/53 
Revised manuscript received 7/26/53 


PSYCHOMETRIKA—VOL. 19, No. 1 
MARCH, 1954 


NOTE ON THE SELECTION OF A PANEL OF JUDGES 
SO AS TO MAXIMIZE PANEL EFFICIENCY* 


STEPHEN Harrison, Norman H. IsHurer, anp M. Lave 
CENTRAL LABORATORIES, GENERAL FOODS CORPORATION 


Attempts to develop improved methods of selecting a panel of judges for 
psychometric work are presented. The applications reported are in the sensory 
field and are limited to the assumption of a unidimensional ability underlying 
the judgments in question. Some aspects of cost efficiency are also considered. 


Introduction 


The situation is quite frequently encountered in psychometric work in 
which a select panel of judges is to be formed from a pool of available indivi- 
duals. The object is to make the selection in such a way as to maximize the 
power of the panel in making statistically reliable discriminations. These 
judgments or discriminations may be in any of a number of areas in the 
psychophysical or psychometric field—e.g., taste testing, color discrimination, 
and so on. Selection of judges is based upon scores which may have been 
derived either from past routine performance or from a special series of screen- 
ing tests. 

Hitherto, very arbitrary standards of selection seem to have been used. 
E.g., it may be decided to select the best 50 per cent of the group, or to 
select those attaining, say, 3/4 of the maximum score in the screening test. 
The present note describes some attempts to develop a more rational approach 
to this problem, and it is hoped that others will be stimulated to carry the 
argument forward at a more rigorous level. 

The applications reported here are in the sensory field (taste testing), but 
the conclusions presumably can be generalized to cover other psychophysical 
and psychometric fields. Only the simplest case will be considered, where it can 
be assumed that the basis of judgment is a unidimensional ability. Further- 
more, this application will assume that each judge can make only one judgment 
on each item. This restriction is dictated by the rapid fatigue associated with 
taste-test work and by the necessity of not impairing motivation of the 
tasters. 


*The writers wish to acknowledge the value of a discussion with Dr. Harold Gulliksen 
of Princeton University and the Educational Testing Service, Princeton; Dr. Frederic Lord, 
also of the Educational Testing Service, contributed some valuable criticisms. The writers’ 
thanks are also due Mr. George Bosy, at the Research Laboratories, Kraft Foods Company, 
where the senior author is currently employed. 
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Some aspects of cost efficiency will also be considered, where the problem 
is to form a select panel which will return the best value for each operating 
dollar expended. In addition, an interesting simplified model will be investi- 
gated, in which the abilities of the pool of judges from which selection is 
to be made are linearly decreasing when ranked in diminishing order of 
magnitude. A frequent observation has been that the corrected screening 
scores of candidate tasters do fall approximately on a straight line when so 
ranked. 


The Problem 


Suppose that there are from 100 to 200 persons from among whom a 
taste-test panel is to be formed. Then the problem is to select those tasters, 
based on screening scores, who will maximize the efficiency of the panel so 
formed. 


The Screening Procedure 


Some of the details described in this section are peculiar to the taste- 
testing problem and not of general application. Their inclusion is necessary 
to make the example clear. 

Panels must be recruited separately for each food product; this does not, 
of course, prevent a given individual from serving on several panels, provided 
he can qualify. A heterogeneous series of samples is assembled which seem to 
cover as well as possible the full gamut of flavor nuances to be expected under 
routine conditions. The samples are presented in groups of three; in each 
such group two of the samples are identical. The task of the subject is to 
identify the odd sample. All samples are, of course, suitably coded to conceal 
the identity of the odd sample. This ‘triangular’ procedure, as it has come 
to be called, is associated with a chance expectancy of 1/3 and is equivalent 
to a multiple-choice question with three alternative answers. A large number 
of such triangular sets are administered over a period of several weeks. For 
details of administration, the reader is referred to Bengtsson and Helm 
(1), Harrison and Elder (8), and Peryam and Swartz (9). 

This procedure ignores the whole question of flavor-dimensionality. 
Presumably a more rigorous approach to the problem would be through the 
use of such mathematical tools as multiple-factor analysis, item analysis, and 
discriminant functions (2, 3, 6, 7, 10), in order to establish the flavor-di- 
mensionality of a given food product. This would involve more extensive 
research than the authors have been prepared to undertake, especially as it 
is far from clear that this would lead to any large gain in precision in selecting 
tasters. The screening method used, then, gives a single measure of ability 
for each taster, without giving any indication of how his ability is distributed 
among the various flavor areas. This restriction somewhat modifies the 
conclusion about the number of persons who ought to be selected for optimum 
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panel efficiency. No attempt was made to correct scores for item-difficulty, 
since there seems to be serious doubt as to whether the increase in precision 
so obtained is sufficient to justify the computational labor. (4, 5). 


Treatment of Data 


The percentage of times the odd sample was correctly selected is recorded 
for each judge, this figure representing the raw score of his ability. These 
scores are corrected so as to eliminate the expected number of correct selec- 
tions due to chance in each case, using the simple formula derived from 
Guilford (4, 445), 


— C) 
~ 100-—C ’ 
where S = percentage score corrected for chance expectation, 


R = raw percentage score, and 
C = percentage score expected by chance. 


In this case, of course, C = 33 1/3 per cent. This correction amounts to 
a simple linear transformation which in no way alters the relative scores. 
An example of the corrected screening scores of 15 participants is shown in 
Table 1. 


TABLE 1 


Tasters’ Scores Ranked in Decreasing Order of Magnitude, After 
Correction for Chance Expectancy 


Rank Score 
1 70% 
2 64% 
3 59% 
4 538% 
5 47% 
6 47% 
7 37% 
8 29% 
9 21% 

10 14% 
11 14% 
12 10% 
13 7% 
14 (-—1%) 
15 (—5%) 


Efficiency of a panel is defined in terms of the probability of that panel’s 
detecting significant differences between samples presented to it. This will 
clearly be a function of two variables, namely, the mean ability of the panel 
and the available degrees of freedom. 
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Let the number of tasters who are eventually selected from the N indi- 
viduals given the screening tests be designated by n. Now imagine, for simplic- 
ity, that subsequent routine testing employing this select panel will be of 
some non-parametric variety such as the triangle tests upon which the 
individuals were screened. Then the efficiency of the select panel in the sense 
defined may be measured by the size of the critical ratio obtained in any given 
instance. This critical ratio will be given by 


R’-—C 


Efficiency = Critical Ratio = ——————. , 
— C) 
n 


where n is the size of the panel and R’ is the raw percentage score obtained 
by this panel. We now make two simplifying assumptions: 


(1) That the ability of a taster to detect differences is proportional to 
his corrected screening score. The mean ability of the n selected persons 
will then be given by 

LS. / n. 


Since the highest corrected screening score was seldom above 70 per cent, 
it was felt that this assumption would not lead to large distortions. 

(2) That the observed difference (R’ — C) in any given subsequent 
routine test would be proportional to the mean ability of the panel, i.e., 


R’-—-C=k) 
i=1 
where k is a suitable constant. Hence: 
i=1 i=1 


n 


Efficiency = 


It is assumed that our tasters are ranked in order of screening score, so that 
S, corresponds to the man with the highest score. It can now be determined 
how the efficiency is related to the select panel size n. Since in any one instance, 
the term kV C(100 — C) will be constant, it may be dropped without alter- 
ing the relative efficiency obtained. Using the example set forth in Table 1, 
then, 

(1) If a one-man panel is formed with the best individual, 


Efficiency (E) = 70//1 = 70; 
(2) Taking the two best men, : 
E = (70 + 64)/V/2 = 91.5. 


| 
S; 
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This procedure was repeated until all N of the original persons screened 
were included. The upper curve in Figure 1 shows the relation between 
panel size and efficiency. It will be seen that the efficiency passes through a 
maximum where n = 8. It would, therefore, appear that selecting those persons 
with the eight top screening scores will yield a select panel with maximum 
efficiency. 
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Relationship of Relative Efficiency to Size of Select Panel 


The reliability of this estimate of the maximum value is limited by the 
reliability of the original scores, particularly the top taster’s score. All scores 
should therefore be based upon large numbers of judgments so that they are 
reasonably reliable. 

There are several reasons why it might be desired to take one or two 
additional persons into our select panel: 


(1) Typically, efficiency diminishes more rapidly to the left of the maxi- 
mum than to the right; it is presumably safer, therefore, to err on the side of 
having too many tasters than too few. 

(2) It has been assumed, in effect, that the taste ability of our judges 
is unidimensional, being represented by their over-all screening scores. As 
pointed out above, this will certainly not be the case. Consequently, added 
tasters may contribute more to the efficiency than the approach has indi- 
cated just because tasters may tend to complement one another in their 
abilities. This again would suggest that the derived maximum value may be 
a little too small. 


| 
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(3) It must be pointed out that the dispersion of screening scores ob- 
tained does not provide an unbiased estimate of the corresponding population 
dispersion, owing to the operation of sampling errors. These have the effect 
of inflating the observed screening scores of the top half of the group and 
depressing the scores of the lower half. The seriousness of this effect depends 
(inversely) upon the number of triangular items included in the screening 
test. Again, the effect will be to cause the number of tasters giving maximum 
panel efficiency to be somewhat too low, since sampling errors are causing 
the men with the higher scores to appear better than they probably are. 

The authors have repeatedly observed that a set of ranked screening 
scores, represented graphically as a bar diagram, form an approximately 
linear “staircase.’’ Although no reason why a set of scores should conform 
to such a simple rational function was apparent, it was decided to investigate 
the dependence of efficiency upon n if it were assumed that the scores ranked 
in order of magnitude do fall on a straight line. 

>_S; can now be replaced by an arithmetical progression, 


5 (28: — — Id}, 


where 


S, is the highest screening score obtained in the group and 
dis the common (negative) difference in score between any two contiguous 
persons. 


We then have 


_ n{28, — (n — 
2V nq Vn 


Taking the first derivative of with respect to n, we have 


Equating to zero and solving for n, we find 


E 


(s, 5 dn = 0, 


1 
n= 3 (2 +1). 


The nth person in such a selected group will clearly have a screening score 
of S, = S, — (n — 1)d. Expressed as a fraction of the best taster’s score, 
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this will be 
S,_ Si—(n—1)d_ 
8; 


Substituting for n, we have 


1 (28, 
s, [5 (+1) - 1 


The value of this conclusion is critically dependent upon the reliability 
of the tasters’ scores. It is therefore desirable for this and other reasons that 
these be reasonably reliable, i.e., be based upon a large number of judgments. 
As the number of persons participating in the screening test becomes large, 
the above value approximates 1/3. The important conclusion follows that 
where the ranked screening scores of a group of persons form an approximately 
linear ‘‘staircase’”’ the most efficient panel is obtained by selecting all persons 
scoring 1/3 or more of the top person’s score. 


Sy _ 


Cost Efficiency 

If it is desired to take the cost of running a taste-test panel into account 
in arriving at an estimate of the number of tasters to select to maximize 
the panel efficiency, the conclusion is altered somewhat. Taste-panel costing 
depends upon two elements: 

C, = fixed cost of setting up a taste test, independent (up to a point) 
of the number of tasters participating, and 

C, = additional cost per taster. 


For a panel of size n, the total cost will therefore be given by 
Co + Cin. 


The cost efficiency may be obtained by dividing this cost by the efficiency 
estimate previously obtained above. We therefore have 
Co + nC, 

pq i=1 
As before, k/V. Pq in any given instance will be a constant multiplier in- 
dependent of n, so 


Cost Efficiency = 


Cost Efficiency (£,) = Lo + 


The panel efficiency for the data given in Table 1 is shown plotted against 
n in Figure 1. The lower two curves in that figure correspond to two different 
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cost-situations, one in which C,/C,; = 10 and the other where C,/C, = 20. 
C, has been put equal to unity. Comparing the two measures of efficiency 
discussed, the one ignoring cost and the other taking cost into account, it 
may be noted that 

(1) Taking cost into account reduces the panel size. This is to be expected 
since each additional taster included in the select panel costs money. 

(2) When C, = 0, then the cost efficiency merely becomes the inverse 
of the efficiency disregarding cost. 

(3) When C, = 0, _ 


t=1 


E, = 


In other words, cost rises so sharply that in all cases the most efficient panel 
would be a one-man panel! But in order to obtain statistically significant 
results, this man would have to make a number of repeat judgments. In 
routine testing work, the authors generally have been reluctant to ask tasters 
to judge more than once or twice a day, so that in this situation it would be 
necessary to increase panel size in the face of less economical operation. 

When tasters’ ranked screening scores form an approximately linear 
“staircase” the rational function for cost efficiency is given by 


Co + nC, 
we {28, — (n — 1)d} 


Cost Efficiency = E, = 
Wa 


Proceeding as before, differentiating with respect to n, equating to zero, and 
solving for n, we have 


Vn (28, — m — nay 


_k 
[3(2S, + d)n 3 dn*\(Cy + 


Vn {28, — (n — | 
= C, dn? + {C,(28, + d) + 3C.d}n — C,(28, + d) = 0. 


The roots of this quadratic are given by 


~{(28 4 1) + 3 + 1) + 10( 28: + 1) + 


2 


; 2 
j 
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Only the positive root is meaningful. It will be seen that the value of n 
depends upon two quantities only, C)/C; and (2S,/d + 1). The first quantity 
is therefore simply the ratio of the two component costs, and the second 
quantity may be considered equal to 2N, where N is a continuous measure 
of the limiting number of tasters available for a select panel; tasters scoring 
below chance expectancy in the screening test are assumed to be excluded. 
So, we can put 


Now, where S, = screening score of the nth taster, 


S, = 8, (n 1)d, 


and 


(N — 4)d 


Finally, substituting in this equation for n, we get 


5, + sv 49 +30, 


N-} 


This fraction marks the dividing point between tasters who should be included 
in, or excluded from, our select panel. Tasters having screening scores above 
this fraction of the best taster’s screening score qualify for inclusion, whereas 
those scoring below should be excluded, if the most cost-efficient select panel 
is to be formed. 

A series of different values for (NV) and C,/C, were substituted in the 
above equation and the value of S,/S, calculated in each case, in order to 
draw some heuristic conclusions about the dependence of S,/S, upon these 
two parameters. 

The resulting family of curves is shown in Figure 2. It can be seen, for 
example, that when N = 20 to 50, and C,/C, is in the region of 10 to 20, 
S,/S, = 3/4; ie., in these circumstances all tasters scoring 3/4 or more of 
the top screening score should be selected for the panel in order to maximize 
cost efficiency. Should these persons be too few to give reasonable sensitivity, 
the panel size may be increased, but not beyond the point where S,/S, = 1/3, 
since this would give the most efficient panel irrespective of cost. 

It must be pointed out that the solution to the maximum problem treated 
in this note would have been much simpler but for a restriction imposed by 
the nature of the psychophysical task. This restriction is that in taste-test 
work under routine conditions, it is not feasible to have each taster evaluate 
a group of samples more than once. Any attempt to make him do so regularly 
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results in fatigue and loss of motivation. If it were not for this restriction, 
then obviously the most efficient evaluation procedure would be to select the 
best man and subject him to a full-fledged psychophysical session with 
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the same group of samples presented a sufficient number of times to secure 
the desired degree of sensitivity and statistical assurance. Such might be 
the case in the fields of vision and audition. 
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BOOK REVIEWS 


CuiypvE H. Coomss. A Theory of Psychological Scaling. Ann Arbor: University of Michigan 
Press, 1952, pp. vi + 94. 


At the outset of this monograph, Coombs expresses his dissatisfaction with what 
might be called the traditional and current theory of psychological measurement. It is 
characteristic of this theory, Coombs argues, to build strong mathematical models. Further- 
more, the mode] builders then proceed to make assumptions about behavior that will adapt 
the behavior to the mathematical model. If the observations of behavior (data) fail to fit 
the model, then the statistical notions of error theory, random variation, and so forth, are 
used to account for or explain away the discrepancies. It is the data that suffer, not the 
theory or mathematical model. 

Coombs, on the other hand, believes that “‘. . . the integrity of the data [should] be 
maintained and the level of measurement adapted to the information in the data’”’ (p. 9). 
Since he also believes that the information contained in the data is very crude, he contends 
that only weak postulates can be expected to be satisfied by the data. It is necessary, 
therefore, to build mathematical models based upon weak postulates if behavior is to be 
described adequately and without resort to the notions of error theory and random varia- 

[ tion. ‘‘To impose stronger systems in spite of data is to build an actuarial science at the 
\ Possible cost of a science of individual behavior’ (p. 9). 

This is the basic issue, as Coombs sees it, between the theory of scaling that he de- 
velops and the traditional theory of psychological measurement. A further contrast, 
however, can be made between the two approaches. Psychometric theory, he argues, has 
had as its objective the construction of scales. It has, therefore, been much concerned over 
the development of techniques and methods which would offer some assurance of success 
in the task of constructing, preferably, unidimensional scales. In this objective Coombs is 
not at all interested. Rather he sets as his objective the discovery of psychological scales. 
If a scale exists for a set of stimuli, this fact alone is of primary psychological significance. 
But, he points out, we cannot conclude that a scale exists if the technique used is such as to 
guarantee its discovery. 

Chapter 1 of the monograph discusses briefly measurement theory both from the 
formal or logical side and from the observational or experimental side. To S. 8. Stevens’ 
description of the ratio, interval, ordinal, and nominal scales, Coombs adds the ordered 
metric and the partially ordered scale. The former is one ‘‘in which the order of the objects 
on a continuum is known, and also the distances between objects may themselves be at 
least partially ordered’’ (p. 3). No precise statement was found defining the partially ordered 
scale, but it would appear from context that by this concept Coombs means that some of the 
stimuli can be ordered on a continuum, but not all. (It is unfortunate that this 94-page 
monograph has no index.) Taking first the scale that holds when the stimuli are regarded as 
elements and then the scale that holds when the distances between the stimuli are regarded 
as elements, Coombs develops a more complete listing of possible types of scales. 

A distinction is made between psychological measurement and psychological scaling, 
although Coombs believes that in the most general sense all scales involve measurement as 
a matter of degree. If the operations of arithmetic are permissible, so that numbers can be 
assigned to the objects or their differences, then measurement is involved. If these operations 
are not permissible, then scaling is involved. ‘‘On this basis, ratio and interval scales are 
both classified as measurement, and the remaining scales, including the ordered metric, 
ordinal scale, partially ordered scale, and nominal scale are classified as scaling’ (p. 5). 
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Since Coombs has only a weak postulational system (no postulates which lead to measure- 
ment are included within the system), his interest is in these latter scales. 

Chapter 2 is concerned with the problem of psychological measurement and scaling. 
Coombs makes the distinction, after Lewin and others, between phenotypic and genotypic 
systems. The genotypic system refers to the underlying, hypothetical basis of behavior. 
The phenotypic system refers to the observed behavior. This distinction is similar to that 
made by Lazarsfeld in his latent attribute analysis, in which the latent attribute is called 
upon to explain manifest behavior. Both the approach of Coombs and that of Lazarsfeld 
stand in marked contrast to Guttman’s scale analysis, which is concerned only with the 
manifest behavior. For Guttman an attitude is a sample of behavior. For Coombs and 
Lazarsfeld an attitude would be “that” which underlies the behavior. 

The postulation of a latent or genotypic “that” raises interesting philosophical and 
methodological questions which cannot be discussed in detail here. This reviewer, for one, 
however, has grave doubts about the usefulness of hypothetical entities that are called upon 
to explain behavior, whether they are called drives, forces, latent attributes, hypothetical 
constructs, or genotypic systems—if these entities themselves cannot be defined in terms 
of observations or operations upon observations. Coombs, on the other hand, although 
concerned with weak postulational systems that will adequately describe behavior, also 
states that his objective is to gain, from the information contained in the phenotypic 
observations, inferences about the underlying genotypic system. The genotypic system, in 
turn, it may be noted, is to be used to explain the phenotypic observations. 

In Chapter 3, Coombs gives his definitions and postulates. Here we find the distinction 
made between Task A and Task B. Task A is concerned with the preference judgments of 
stimuli with the individual as point of reference. Task B is concerned with judgments of 
stimuli with respect to some attribute. For Task B, the individual’s own position on the 
continuum under investigation is irrelevant, i.e., the judgments made are assumed to be 
independent of the location of the judge on the continuum. This is Thurstone’s contention, 
stated in 1939, that judgments of the degree of favorableness and unfavorableness of attitude 
items are independent of the attitudes of the subjects doing the judging. 

The researches of Hinckley and others on this point, in the thirties, seemed conclusive 
in showing that Task B judgments were, in fact, independent of the attitudes of the judges. 
Still, some of us were concerned about the issue. For example, in 1946, this reviewer and 
K. C. Kenney stated: 


We are not satisfied with the evidence on this point. Would similar results 
obtain from judgments derived from those with sympathetic attitudes toward 
fascism and those violently opposed to fascism in the construction of a scale 
measuring attitude toward fascism? And in the case of communist sympa- 
thizers and non-communists in the construction of a scale measuring attitude 
toward communism? When social approval or disapproval attaches to a 
favorable or unfavorable attitude toward an issue, different scale values might 
result from groups with differing attitudes. .. . The research so far, it seems to 
us, also neglects the related problem of ego-involved attitudes and the bearing 
they might have upon scale values of items. 


Now we find that recent research by Hovland and Sherif provides evidence that the 
procedures used in scaling attitude items by the method of equal-appearing intervals ruled 
out the possibility of the data showing the influence of the judges’ attitudes upon the scale 
values obtained. The reason was a simple one, namely, that individuals with extreme atti- 
tudes were incidentally eliminated by a criterion of ‘‘carelessness.’? Hovland and Sherif 
show that it is precisely these “careless” judges who fall at the two extreme ends of the 
continuum. And the scale values of items obtained from the judgments of these extreme 
groups do show the influence of the judges’ positions on the continuum. 

Coombs will have to reconcile the findings of Hovland and Sherif with the assumption 
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he has made concerning the irrelevance of the position of the individual for judgments 
obtained in Task B. Since he makes no pretension of providing a completed theory (there 
are many departures from the position taken in his earlier articles), the necessary modifica- 
tions in his postulational system should be possible. 

The first three chapters, described briefly above, contain the essentials of Coombs’ 
theory. Chapter 4 develops equations for the genotypic parameters and Chapter 5 for the 
phenotypic parameters. Chapter 6 then deals with certain interrelations between the 
genotypic and phenotypic parameters. 

Chapter 7 is concerned with the “unfolding technique,’ a method which Coombs 
believes enables one “‘to go behind the expressed preferences of individuals and to construct 
a model from which their preferences may be derived’’ (p. 56). In this chapter we also find 
the statement that “there is no objective way in the Method of Paired Comparisons of 
testing the consistency of an individual’s judgments” (p. 71). In an earlier article (1950), 
Coombs had defined consistency in terms of the phenotypic system. In the present mono- 
graph, he defined consistency in terms of the genotypic system. If consistency refers to the 
underlying genotypic system, Coombs’ statement regarding a test for consistency might be 
accepted. However, on the phenotypic level, Kendall’s work on paired comparisons, his 
development of a coefficient of consistence, and the x? test of significance of the coefficient 
of consistence would seem applicable. - 

The monograph concludes with Chapter 8, which is concerned with some of the 

implications of the scaling theory developed for Thurstone’s law of comparative judgment. 
Coombs believes that properly there are two such laws, one for Task A and one for Task B; 
i.e., judgments of preference and judgments of attributes, respectively. Again some modifica- 
tion of this section will apparently be necessary to take into account the empirical findings 
of Hovland and Sherif. 

An over-al] judgment of Coombs’ contribution would be an exceedingly difficult task, 
regardless of whether the judgment is made from the point of view of Task A or Task B. 

Without concern for the nature of the scale involved, the Task B judgment of the reviewer 
is that in some respects Coombs falls somewhere near Lazarsfeld, in that both, for example, 
are interested in genotypic or latent systems. In some respects he also falls near Guttman, 
for both are interested in the discovery rather than the construction of scales. In terms of 
techniques used for the collection of data, such as the method of paired comparisons, 
Coombs might be placed near Thurstone. 

Again without concern for the nature of the scale involved, from a Task A point of 
view, the judgment is easier. From a preference point of view, this reviewer belongs at that 
point on a continuum where fall those who like strong postulational systems, the consequent 
operations of arithmetic, and the applications of error theory and random variation. Coombs, 
by choice, has scaled himself at the point where those who prefer weak postulational systems 
fall. Consequently, if a single continuum is involved, Coombs is at one end and the reviewer 
is at the other. 


University of Washington Allen L. Edwards 


KeEnneEtH L. Bran. Construction of Educational and Personnel Tests. New York: McGraw- 
Hill, 1953, pp. viii + 231. 


The author begins with the premise that the majority of teachers and others who are 
faced with the job of constructing and using tests have not had sufficient training in test 
construction. This condition is attributed essentially to inadequate curricula and to a 
lack of textual treatises on the subject of test construction. 

It is specifically stated that “Most individuals who are interested in using test . 


if 
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results, except research workers, do not like to become lost in technical verbiage beyond 
their comprehension.”’ Accordingly, the book is written in a style and manner which may 
not appeal to the highly qualified psychologist but which will be appreciated by the greater 
portion of test constructors. 

The book is not a thorough treatise of the subject of test construction; however, it 
does present many of the essentials of test construction in such a manner that potential 
test constructors with a moderate amount of sophistication will benefit from reading it. 
On the more technical aspects of test contruction, the author makes ample reference to 
sources of information which are of a higher technical level. 

The author presents elementary concepts of measurement in a very understandable 
manner. There is a good discussion of terminology in an attempt to clarify some of the 
related terms such as aptitude, capacity, and talent. There is coverage of such matters 
as the distinction between achievement and aptitude measurement. 

The author fully recognizes the importance of the often neglected steps of planning 
the examination. He recognizes the importance of job analysis and planning for the admin- 
istration and scoring of the test. He points out that validation should be planned for 
while still constructing the test. There is no coverage of the procedures, or mechanics, of 
validation, nor mention of the difficulties and pitfalls involved in attempting to conduct 
a statistically sound validation study. The author presents a number of specific rules for 
constructing true-false, multiple choice, completion, and matching test items. He dis- 
cusses advantages and disadvantages of essay-type tests. He refers to ‘approaching 
reasonable objectivity” in scoring essay-type tests without mentioning the extreme difficulty 
of achieving such objectivity, reasonable or otherwise. 

The book contains a short chapter on performance tests. There is no discussion of 
the cumbersoneness of most performance tests, the time, expense, and difficulty of scoring, 
when used for most jobs other than those such as office machine operators, typists, and 
stenographers. 

The importance of having new test items reviewed by subject matter experts prior 
to use is pointed out. A method of item analysis which any test constructor can use is 
presented. 

In conclusion, it should be stated that the book can serve a worthy purpose. It is 
believed that many potential test constructors who would ordinarily ‘‘shy away’ from 
the more technical references on test construction will have a good “starter’’ in this book 
and will in turn be introduced to more technical references. 


Personnel Research Branch 
Personnel Research and Procedures Division 
The Adjutant General’s Office Rudolph G. Berkhouse 


Witrrip J. Drxon and Frank J. Massey, Jr. An Introduction to Statistical Analysis. 
New York: McGraw-Hill, 1951, pp. x + 370. $4.75. 


Intended for students in various disciplines who desire a basic course in statistics, 
the book Introduction to Statistical Analysis, represents a unique contribution to the litera- 
ture of elementary texts in the field. In a lucid style and with a marked degree of simplicity 
the authors have presented a wealth of material and have logically and coherently developed 
a vast number of concepts in statistical thinking. Probably the most remarkable feature 
of the book is that an intelligent student with a background of but three semesters of high 
school algebra can understand virtually every topic considered. Although authors of 
elementary texts in mathematical statistics have frequently maintained that little formal 
mathematical training is required on the part of prospective readers, Dixon and Massey 
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have probably been the first ones to succeed in their objectives of adapting the content of 
their subject to the level of mathematical maturity of the majority of college students. 

Throughout most of the text the authors have displayed an unusual grasp of the psy- 
chology of learning in terms of a common-sense and intuitive approach to the explanation 
of concepts within the areas of sampling and hypothesis testing—an approach embodying 
a favorable balance of verbalization, geometric representation, and algebraic expression. 
A substantial degree of the pedagogical success achieved is probably due to the fact that. 
the authors have followed to a considerable extent the recommendations made by the 
committee on teaching of statistics of the National Research Council as to the order and 
emphasis of topics. 

Numerous illustrative examples are included that serve to reinforce the expository 
passages within the text. Particularly appealing is the employment of an outline form with 
respect to which explanatory problems are worked. In each of the illustrative examples: 
concerning the testing of hypotheses the steps involved are given in logical order and 
described in a precise manner. Although usually based on ficticious data, the problem 
exercises at the end of each chapter are thought provoking and essential to mastery of 
the fundamental concepts developed in the chapters. (Answers are available from the 
authors upon request.) Several class exercises are included, many of which aim toward 
the experimental verification of sampling distributions. 

For the student of the social sciences the text has much to offer in terms of the attain- 
ment of a broader perspective or familiarity with the possible applications of statistics 
in various fields. Perhaps more important to the social science worker is his exposure to 
certain intangible features underlying the viewpoint of the mathematical statistician. 
In reading this book and in teaching a course organized about it, the reviewer has gained 
the impression of the existence of a certain inherent philosophy in statistical thinking 
not found in most texts written by statisticians specializing in psychology, education, or 
sociology. To some extent this feeling may have been due to the emphasis placed by the 
writers upon such relatively modern developments as a@ and £8 errors, power tests, non- 
parametric statistics, statistical efficiency, and sequential analysis. 

One cannot help being impressed with the twenty-six different tables included within 
the 72 pages of the Appendix. Many of the tables which are not to be found in other texts 
are based on developments in statistical theory that have appeared in the journals since 
1945. At various points in the text examples are included that serve to demonstrate use 
of the tables. 

Only two possible criticisms of the text come to mind. These are based upon reports 
which the reviewer has received from students. First, students consistently have a great 
deal of difficulty in understanding how to use the tables of random numbers. The explana- 
tion given on page 35 appears to be somewhat confusing, if not incomplete. One or two: 
illustrative examples at this point would probably be helpful. Second, several students— 
especially those with limited mathematical maturity—state that they find it hard to grasp 
the central implications of a and 8 errors developed in Chapter 7 and expanded upon in 
Chapter 14. In particular, the explanation regarding the power of a test and the power of 
a critical region against certain alternatives does not satisfy many students. Although 
the authors’ exposition appears to be quite explicit to the reviewer, it may be that the 
incorporation of additional diagrams involving overlapping geometric curves with appro- 
priate designations of a aud 8 in terms of shaded areas or the insertion of two or three 
more illustrative examples would be helpful. Several students have mentioned that it 
was not until they studied the first two or three pages of the Chapter 18 “Sequential 
Analysis” that they began to grasp the import of a and £ errors and to comprehend the 
réle of power tests associated with these errors. 

In addition to the topics cited that are based upon relatively recent developments 
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such conventional subjects as analysis of variance, regression and correlation, analysis of 
covariance, and enumeration statistics are considered. For the psychology student the 
content of the chapter concerning regression and correlation is probably inadequate, since 
discussion is limited to a consideration of two-variable correlation. However, it is not 
reasonable to expect that an introductory text can cover adequately specific points of 
emphasis in the various fields to which statistical methods are applied. 

In any event, the student in the social sciences who is seriously interested in under- 
standing and applying the fundamentals of statistics will do well to read this book. In 
fact, the social scientist who wishes a reference text on a variety of topics within the area 
of statistics should by all means purchase this book for his library, as he will find it to be 
a convenient working aid and an excellent source for periodic review. 


William B. Michael 
The University of Southern California 
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